Real time position sensing of objects

ABSTRACT

Embodiments are directed toward measuring a three dimensional range to a target. A transmitter emits light toward the target. An aperture may receive light reflections from the target. The aperture may direct the reflections toward a sensor that comprises rows of pixels that have columns. The sensor is offset a predetermined distance from the transmitter. Anticipated arrival times of the reflections on the sensor are based on the departure times and the predetermined offset distance. A portion of the pixels are sequentially activated based on the anticipated arrival times. The target&#39;s three dimensional range measurement is based on the reflections detected by the portion of the pixels.

CROSS-REFERENCE TO RELATED APPLICATIONS

This Utility Patent Application is a Continuation of U.S. patentapplication Ser. No. 16/398,139 filed on Apr. 29, 2019, now U.S. Pat.No. 10,502,815 issued on Dec. 10, 2019, which is a Continuation of U.S.Patent Application Serial No. 15/694,532 filed on Sep. 1, 2017, now U.S.Pat. No. 10,274,588 issued on Apr. 30, 2019, which is a Continuation ofU.S. patent application Ser. No. 15/384,227 filed on Dec. 19, 2016, nowU.S. Pat. No. 9,753,126 issued on Sep. 5, 2017, which is based on apreviously filed U.S. Provisional Patent Application Ser. No. 62/386,991filed on Dec. 18, 2015, and U.S. Provisional Patent Application Ser. No.62/391,637 filed on May 3, 2016, and U.S. Provisional Patent ApplicationU.S. Ser. No. 62/495,667 filed on Sep. 19, 2016, the benefits of thefiling dates of which are hereby claimed under 35 U.S.C. § 119(e) and §120 and the contents of which are each incorporated in entirety byreference.

TECHNICAL FIELD

The present invention relates generally to three-dimensional trackingsystems and, more particularly, but not exclusively, to employingsequential pixel beam scans in highly compact laser-based projectionsystems.

BACKGROUND

Tracking systems may be employed to track a position and/or a trajectoryof a remote object, such as an aircraft, a missile, a drone, aprojectile, a baseball, a vehicle, or the like. The systems may trackthe remote object based on detection of photons, or other signals,emitted and/or reflected by the remote object. The tracking systems mayilluminate the remote object with electromagnetic waves, or light beams,emitted by the tracking systems. The tracking systems may detect aportion of light beams that are reflected, or scattered, by the remoteobject. The tracking systems may suffer from one or more of undesirablespeed, undesirable accuracy, or undesirable susceptibility to noise.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an embodiment of an exemplary environment in which variousembodiments of the invention may be implemented;

FIG. 2 illustrates an embodiment of an exemplary mobile computer thatmay be included in a system such as that shown in FIG. 1;

FIG. 3 shows an embodiment of an exemplary network computer that may beincluded in a system such as that shown in FIG. 1;

FIG. 4 illustrates an embodiment of a three-dimensional perspective viewof an exemplary transmit and receive (T_(x)-R_(x)) system;

FIG. 5 shows an embodiment of a two-dimensional view of an exemplaryreceiving system;

FIG. 6 illustrates an embodiment of a three-dimensional perspective viewof an exemplary two-dimensional position tracking receiver system;

FIG. 7 shows an embodiment of an exemplary strobed search light (SSL)with voxel-sized trigger beam (TB) that is scanning across athree-dimensional surface;

FIG. 8 illustrates an embodiment of an exemplary narrowly-collimatedvoxel-sized trigger beam that is surrounded by an exemplary flying spotstyle SSL;

FIG. 9 shows an embodiment of an exemplary narrowly-collimatedvoxel-sized trigger beam that is trailed by an exemplary flying spotstyle SSL;

FIG. 10 illustrates an embodiment of an exemplary three-dimensionalperspective view of a fast tracking system that employs three exemplaryone-dimensional real time sensors to track an exemplary flying spotstyle SSL;

FIG. 11 shows another embodiment of a three-dimensional perspective viewof an exemplary fast tracking system that employs three exemplaryone-dimensional real time sensors to track an exemplary flying spotstyle SSL;

FIG. 12 illustrates another embodiment of a three-dimensionalperspective view of an exemplary fast tracking system that employs threeexemplary one-dimensional real time sensors to track an exemplary flyingspot style SSL;

FIG. 13 shows an embodiment of an exemplary two-dimensional view of anexemplary stereo receiving system;

FIG. 14 illustrates an embodiment of an exemplary retro-reflectivetarget that splits and reflects an exemplary scanning beam into twoexemplary separate beams;

FIG. 15 shows an embodiment of a three-dimensional perspective view ofan exemplary cubic retro-reflective facet that splits and reflects anexemplary scanning beam;

FIG. 16 illustrates an embodiment of three-dimensional perspective viewof an exemplary retro-reflective target that splits and reflects anexemplary scanning beam into two exemplary separate beams towardexemplary receivers on an exemplary vehicle;

FIG. 17 shows another embodiment of a two-dimensional perspective viewof an exemplary leading vehicle that employs exemplary retro-reflectivetargets that split and reflect an exemplary scanning beam into twoseparate exemplary beams toward exemplary receivers of an exemplaryfollowing vehicle;

FIG. 18 illustrates a logical flow diagram showing an exemplary processfor dynamically switching from an exemplary triangulation mode to anexemplary time of flight (TOF) mode;

FIG. 19 illustrates an embodiment of a three-dimensional perspectiveview of an exemplary scan mirror that prevents an exemplary pincushiondistortion;

FIG. 20 shows an embodiment of an exemplary sensor grid that hasexemplary pixel and row geometries that match an exemplary fast linescan trajectories to prevent an exemplary optical distortion;

FIG. 21 illustrates an embodiment of a two-dimensional perspective viewof an exemplary scanning system with an exemplary multi-focus cameraarray that provides exemplary overlapping fields of view;

FIG. 22 shows an embodiment of a two-dimensional perspective view of anexemplary scanning system with an exemplary multi-focus camera arrayemploying an exemplary wide angle field of view;

FIG. 23 illustrates an embodiment of a two-dimensional perspective viewof an exemplary scanning system with an exemplary multi-focus cameraarray employing an exemplary narrow angle field of view;

FIG. 24 illustrates an embodiment of an exemplary four-transistorphotodiode pixel;

FIG. 25 show an embodiment of an exemplary two-transistor photodiodepixel;

FIG. 26 shows an embodiment of an exemplary flashed illuminationemploying exemplary wave-front color separation;

FIG. 27 illustrates an embodiment of an exemplary cascaded trigger pixelsystem that employs exemplary separate sense lines to sequentiallycapture various exemplary color-separated and time-separated components;

FIG. 28 shows an embodiment of an exemplary flash-triggeredfour-transistor photodiode pixel that employs an exemplary sense line tosequentially capture various exemplary color-separated andtime-separated components;

FIG. 29 shows an embodiment of a two-dimensional perspective view of anexemplary stereo pair of exemplary triangulating LIDAR receivers thatdetect exemplary reflecting light waves of exemplary tracer bullets,wherein pixels in each of the receivers are, for each individual emittedtracer bullet, individually anticipatorily activated to synchronize eachactive ON period of each individual pixel with an anticipated returntime and anticipated pixel location of each possible reflection for eachpossible range and location along a path of travel of the individualemitted tracer bullet;

FIG. 30 illustrates an embodiment of a two-dimensional perspective viewof an exemplary LIDAR triangulating transmit and receive (T_(x)-R_(x))system with exemplary disparity proportional to time of flight;

FIG. 31 shows an embodiment of a two-dimensional perspective view ofexemplary spatial range selection;

FIG. 32 illustrates an embodiment of a three-dimensional perspectiveview of exemplary three-dimensional range selection;

FIG. 33 shows an embodiment of a two-dimensional perspective view of anexemplary transmit and receive (T_(x)-R_(x)) system that managesexemplary pixels to detect and differentiate between exemplary “earlybirds” and “late stragglers”;

FIG. 34 illustrates an embodiment of a two-dimensional perspective viewof an exemplary transmit and receive (T_(x)-R_(x)) system that employsexemplary background elimination and foreground elimination;

FIG. 35 shows an embodiment of a two-dimensional perspective view of anexemplary transmit and receive (T_(x)-R_(x)) system that Z-locks on anexemplary surface to be mapped;

FIG. 36 illustrates an embodiment of an exemplary two-dimensional LIDARsystem that uses exemplary small range expectation and locationdeterminism to range lock and shutter exemplary successive incomingphotons;

FIG. 37 shows an embodiment of an exemplary assisted stereo scanningsystem with an exemplary fast sliding register-logic epipolar parallelstereo matching system;

FIG. 38 shows an embodiment of a two-dimensional perspective view of anexemplary transmit and receive (T_(x)-R_(x)) system that sequentiallysets and activates exemplary pixels;

FIG. 39 illustrates an embodiment of a two-dimensional perspective of anexemplary transmit and receive (T_(x)-R_(x)) system that employssuccessive exemplary rays to obtain exemplary corresponding disparities;

FIG. 40 shows an embodiment of a two-dimensional perspective of anexemplary transmit and receive (T_(x)-R_(x)) system that employsexemplary color coding to prevent ambiguity and to increase exemplaryscanning rates;

FIG. 41 illustrates an embodiment of an exemplary “twitchy pixel” thatemploys exemplary real-time pixel and column shuttering and dynamicsensitivity adjustment;

FIG. 42 shows an embodiment of exemplary activation and gain controlcircuitry built into an exemplary pixel;

FIG. 43 illustrates an embodiment of exemplary gain control circuitrybuilt into an exemplary column sense line amplifier;

FIG. 44 shows an embodiment of a three-dimensional perspective view ofan exemplary transmit and receive (T_(x)-R_(x)) system that employsexemplary light blades and an exemplary SPAD array sensor;

FIG. 45 illustrates an embodiment of a two-dimensional perspective viewof an exemplary active column-gated SPAD array sensor;

FIG. 46 shows an embodiment of exemplary choreographed successive SPADpixel column activation;

FIG. 47 illustrates an embodiment of a two-dimensional perspective viewof an exemplary transmit and receive (T_(x)-R_(x)) system that employsan exemplary series of light blades and an exemplary SPAD array thatcaptures exemplary reflections of the light blades; and

FIG. 48 shows an embodiment of a two-dimensional perspective view of anexemplary SPAD array that captures an exemplary series of exemplarylight blades at exemplary positions that have exemplary disparity deltasthat vary as an exemplary function of distance between an exemplaryreceiver and an exemplary transmitter and as an exemplary function ofZ-range of a target object.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Various embodiments now will be described more fully hereinafter withreference to the accompanying drawings, which form a part hereof, andwhich show, by way of illustration, specific embodiments by which theinvention may be practiced. The embodiments may, however, be embodied inmany different forms and should not be construed as limited to theembodiments set forth herein; rather, these embodiments are provided sothat this disclosure will be thorough and complete, and will fullyconvey the scope of the embodiments to those skilled in the art. Amongother things, the various embodiments may be methods, systems, media, ordevices. Accordingly, the various embodiments may take the form of anentirely hardware embodiment, an entirely software embodiment, or anembodiment combining software and hardware aspects. The followingdetailed description is, therefore, not to be taken in a limiting sense.

Throughout the specification and claims, the following terms take themeanings explicitly associated herein, unless the context clearlydictates otherwise. The phrase “in one embodiment” as used herein doesnot necessarily refer to the same embodiment, though it may.Furthermore, the phrase “in another embodiment” as used herein does notnecessarily refer to a different embodiment, although it may. Thus, asdescribed below, various embodiments of the invention may be readilycombined, without departing from the scope or spirit of the invention.

In addition, as used herein, the term “or” is an inclusive “or”operator, and is equivalent to the term “and/or,” unless the contextclearly dictates otherwise. The term “based on” is not exclusive andallows for being based on additional factors not described, unless thecontext clearly dictates otherwise. In addition, throughout thespecification, the meaning of “a,” “an,” and “the” include pluralreferences. The meaning of “in” includes “in” and “on.”

As used herein, the terms “photon beam,” “light beam,” “electromagneticbeam,” “image beam,” or “beam” refer to a somewhat localized (in timeand space) beam or bundle of photons or electromagnetic (EM) waves ofvarious frequencies or wavelengths within the EM spectrum. An outgoinglight beam is a beam that is transmitted by various ones of the variousembodiments disclosed herein. An incoming light beam is a beam that isdetected by various ones of the various embodiments disclosed herein.

As used herein, the terms “light source,” “photon source,” or “source”refer to various devices that are capable of emitting, providing,transmitting, or generating one or more photons or EM waves of one ormore wavelengths or frequencies within the EM spectrum. A light orphoton source may transmit one or more outgoing light beams. A photonsource may be a laser, a light emitting diode (LED), a light bulb, orthe like. A photon source may generate photons via stimulated emissionsof atoms or molecules, an incandescent process, or various othermechanism that generates an EM wave or one or more photons. A photonsource may provide continuous or pulsed outgoing light beams of apredetermined frequency, or range of frequencies. The outgoing lightbeams may be coherent light beams. The photons emitted by a light sourcemay be of various wavelengths or frequencies.

As used herein, the terms “photon detector,” “light detector,”“detector,” “photon sensor,” “light sensor,” or “sensor” refer tovarious devices that are sensitive to the presence of one or morephotons of one or more wavelengths or frequencies of the EM spectrum. Aphoton detector may include an array of photon detectors, such as anarrangement of a plurality of photon detecting or sensing pixels. One ormore of the pixels may be a photosensor that is sensitive to theabsorption of at least one photon. A photon detector may generate asignal in response to the absorption of one or more photons. A photondetector may include a one-dimensional (1D) array of pixels. However, inother embodiments, photon detector may include at least atwo-dimensional (2D) array of pixels. The pixels may include variousphoton-sensitive technologies, such as one or more of active-pixelsensors (APS), charge-coupled devices (CCDs), Single Photon AvalancheDetector (SPAD) (operated in avalanche mode or Geiger mode),photovoltaic cells, phototransistors, twitchy pixels, or the like. Aphoton detector may detect one or more incoming light beams.

As used herein, the term “target” is one or more various 2D or 3D bodiesthat reflect or scatter at least a portion of incident light, EM waves,or photons. For instance, a target may scatter or reflect an outgoinglight beam that is transmitted by various ones of the variousembodiments disclosed herein. In the various embodiments describedherein, one or more photon sources may be in relative motion to one ormore of photon detectors and/or one or more targets. Similarly, one ormore photon detectors may be in relative motion to one or more of photonsources and/or one or more targets. One or more targets may be inrelative motion to one or more of photon sources and/or one or morephoton detectors.

As used herein, the term “disparity” represents a positional offset ofone or more pixels in a sensor relative to a predetermined position inthe sensor. For example, horizontal and vertical disparities of a givenpixel in a sensor may represent horizontal and vertical offsets (e.g.,as indicated by row or column number, units of distance, or the like) ofthe given pixel from a predetermined position in the sensor (or anothersensor). The disparities may be measured from a center, one or moreedges, one or more other pixels, or the like in the sensor (or anothersensor). In other embodiments, disparity may represent an angle. Forexample, a transmitter may emit a beam at an angle α, and the sensor mayreceive a reflection of the beam at an angle β through an aperture. Thedisparity may be measured as the difference between 180° and the sum ofthe angles α and β.

The following briefly describes embodiments of the invention in order toprovide a basic understanding of some aspects of the invention. Thisbrief description is not intended as an extensive overview. It is notintended to identify key or critical elements, or to delineate orotherwise narrow the scope. Its purpose is merely to present someconcepts in a simplified form as a prelude to the more detaileddescription that is presented later.

Briefly stated, various embodiments are directed to measuring a distanceto a target that reflects light from a transmitter to a receiver. Thereceiver may be offset from the transmitter by a predetermined distanceand may include a sensor that has one or more rows of pixels. Thepredetermined offset distance between the transmitter and the receiverenables speculatively activating, in sequence, pixels in a row of thesensor. The sequential speculative activation may be based onanticipating, for each instance of time following emission of the lightfrom the transmitter, which one or more pixels would receive areflection if the target were at a corresponding distance from thetransmitter. In one or more of the various embodiments, if one or morespeculatively activated pixels do not capture a reflection of the lightfrom the transmitter, the sequence of speculative activation continuesin anticipation of the target being at a further distance. In some ofthe various embodiments, responsive to one or more speculativelyactivated pixels capturing a reflection of the light from thetransmitter, the distance to the target may be determined based on theposition of the one or more speculatively activated pixels.

In one or more of the various embodiments, each speculatively activatedpixel is activated for a duration that is based on the offset distance,an angle at which the transmitter emitted the light, and an amount oftime that has passed since the transmitter emitted the light. Ananticipated path of reflection to a speculatively activated pixel fromthe distance that corresponds to the speculatively activated pixel maydefine an angle with the sensor. The sequence of speculative activationmay progress such that, over time, the defined angle approaches thedifference between 180° and the angle at which the transmitter emittedthe light. In some of the various embodiments, as the anticipated pathof reflection becomes more parallel to the path along which thetransmitter emitted the light, each speculatively activated pixel maycorrespond to a greater range of potential distances to the target(e.g., the duration of activation of each speculatively activated pixelmay increase as the sequence of speculative activation progresses).

In one or more of the various embodiments, the light emitted by thetransmitter may be a continuous beam that is scanned across a field ofview of the sensor. In some of the various embodiments, the lightemitted by the transmitter may form a blade that has a longitudinaldimension that is perpendicular (or more perpendicular than parallel) tothe one or more rows of the sensor and also perpendicular (or moreperpendicular than parallel) to the path along which the transmitteremitted the light. Speculatively activated pixels in multiple rows inthe sensor may each capture a reflection of the blade from the target.In some embodiments, each column of pixels in the sensor may report tothe same column sense line.

In one or more of the various embodiments, responsive to the sequence ofspeculative activation progressing to a point where each speculativelyactivated pixel corresponds to a range of potential distances thatexceeds a threshold value, one or more different distance measuringmodes may be employed instead of speculative sequential activation. Insome of the various embodiments, where multiple sensors are employed andone or more of the multiple sensors reports capture of a reflection fromthe target, the one or more different distance measuring modes may beemployed responsive to one or more of the multiple sensors failing tocapture a reflection from the target. In some embodiments, responsive torepeating the sequence of sequential activation a certain number oftimes without capturing a reflection from the target, the one or moredifferent distance measuring modes may be employed.

In one or more of the various embodiments, the one or more differentdistance measuring modes may employ an intense (e.g., compared to anamplitude of the light emitted for the speculative activation sequence)pulsed beam emitted by the transmitter. In some of the variousembodiments, each pulse of the pulsed beam may include a distinct colorcompared to that of an immediately prior and an immediately subsequentpulse. In some embodiments, the one or more different distance measuringmodes may employ fast amplitude or frequency modulation of the pulses.In one or more of the various embodiments, the one or more differentdistance measuring modes may include determining the distance to thetarget based on time of flight of a burst of light emitted by thetransmitter. In some of the various embodiments, the one or moredifferent distance measuring modes may employ one or more differentsensors (e.g., LIDAR or radar sensors).

In one or more of the various embodiments, the pixels in the sensor mayinclude Single Photon Avalanche Diodes (SPADs). In some of the variousembodiments, the pixels in the sensor may report capture of a reflectionvia one or more of a high sensitivity sensing amplifier or a sourcefollower connected to a photodiode in each of the pixels (e.g., “twitchypixels”).

Illustrated Operating Environment

FIG. 1 shows exemplary components of one embodiment of an exemplaryenvironment in which various exemplary embodiments of the invention maybe practiced. Not all of the components may be required to practice theinvention, and variations in the arrangement and type of the componentsmay be made without departing from the spirit or scope of the invention.As shown, system 100 of FIG. 1 includes network 102, photon transmitter104, photon receiver 106, target 108, and tracking computer device 110.In some embodiments, system 100 may include one or more other computers,such as but not limited to laptop computer 112 and/or mobile computer,such as but not limited to a smartphone or tablet 114. In someembodiments, photon transmitter 104 and/or photon receiver 106 mayinclude one or more components included in a computer, such as but notlimited to various ones of computers 110, 112, or 114.

System 100, as well as other systems discussed herein, may be asequential-pixel photon projection system. In at least one embodimentsystem 100 is a sequential-pixel laser projection system that includesvisible and/or non-visible photon sources. Various embodiments of suchsystems are described in detail in at least U.S. Pat. No. 8,282,222,U.S. Pat. No. 8,430,512, U.S. Pat. No. 8,696,141, U.S. Pat. No.8,711,370, U.S. Patent Publication No. 2013/0300,637, and U.S. PatentPublication No. 2016/0041266. Note that each of the U.S. patents andU.S. patent publications listed above are herein incorporated byreference in the entirety.

Target 108 may be a three-dimensional target. Target 108 is not anidealized black body, i.e. it reflects or scatters at least a portion ofincident photons. As shown by the velocity vector associated with photonreceiver 106, in some embodiments, photon receiver 106 is in relativemotion to at least one of photon transmitter 104 and/or target 108. Forthe embodiment of FIG. 1, photon transmitter 104 and target 108 arestationary with respect to one another. However, in other embodiments,photon transmitter 104 and target 108 are in relative motion. In atleast one embodiment, photon receiver 106 may be stationary with respectto one or more of photon transmitter 104 and/or target 108. Accordingly,each of photon transmitter 104, target 108, and photon receiver 106 maybe stationary or in relative motion to various other ones of photontransmitter 104, target 108, and photon receiver 106. Furthermore, asused herein, the term “motion” may refer to translational motion alongone or more of three orthogonal special dimensions and/or rotationalmotion about one or more corresponding rotational axis.

Photon transmitter 104 is described in more detail below. Briefly,however, photon transmitter 104 may include one or more photon sourcesfor transmitting light or photon beams. A photon source may includephoto-diodes. A photon source may provide continuous or pulsed lightbeams of a predetermined frequency, or range of frequencies. Theprovided light beams may be coherent light beams. A photon source may bea laser. For instance, photon transmitter 104 may include one or morevisible and/or non-visible laser source. In one embodiment, photontransmitter 104 includes at least one of a red (R), a green (G), and ablue (B) laser source to produce a RGB image. In some embodiment, photontransmitter includes at least one non-visible laser source, such as anear-infrared (NIR) laser. Photon transmitter 104 may be a projector.Photon transmitter 104 may include various ones of the features,components, or functionality of a computer device, including but notlimited to mobile computer 200 of FIG. 2 and/or network computer 300 ofFIG. 3.

Photon transmitter 104 also includes an optical system that includesoptical components to direct, focus, and scan the transmitted oroutgoing light beams. The optical systems aim and shape the spatial andtemporal beam profiles of outgoing light beams. The optical system maycollimate, fan-out, or otherwise manipulate the outgoing light beams. Atleast a portion of the outgoing light beams are aimed at and arereflected by the target 108. In at least one embodiment, photontransmitter 104 includes one or more photon detectors for detectingincoming photons reflected from target 108, e.g., transmitter 104 is atransceiver.

Photon receiver 106 is described in more detail below. Briefly, however,photon receiver 106 may include one or more photon-sensitive, orphoton-detecting, arrays of sensor pixels. An array of sensor pixelsdetects continuous or pulsed light beams reflected from target 108. Thearray of pixels may be a one dimensional-array or a two-dimensionalarray. The pixels may include SPAD pixels or other photo-sensitiveelements that avalanche upon the illumination one or a few incomingphotons. The pixels may have ultra-fast response times in detecting asingle or a few photons that are on the order of a few nanoseconds. Thepixels may be sensitive to the frequencies emitted or transmitted byphoton transmitter 104 and relatively insensitive to other frequencies.Photon receiver 106 also includes an optical system that includesoptical components to direct, focus, and scan the received, or incoming,beams, across the array of pixels. In at least one embodiment, photonreceiver 106 includes one or more photon sources for emitting photonstoward the target 108 (e.g., receiver 106 includes a transceiver).Photon receiver 106 may include a camera. Photon receiver 106 mayinclude various ones of the features, components, or functionality of acomputer device, including but not limited to mobile computer 200 ofFIG. 2 and/or network computer 300 of FIG. 3.

Various embodiment of tracking computer device 110 are described in moredetail below in conjunction with FIGS. 2-3 (e.g., tracking computerdevice 110 may be an embodiment of mobile computer 200 of FIG. 2 and/ornetwork computer 300 of FIG. 3). Briefly, however, tracking computerdevice 110 includes virtually various computer devices enabled toperform the various tracking processes and/or methods discussed herein,based on the detection of photons reflected from one or more surfaces,including but not limited to surfaces of target 108. Based on thedetected photons or light beams, tracking computer device 110 may alteror otherwise modify one or more configurations of photon transmitter 104and photon receiver 106. It should be understood that the functionalityof tracking computer device 110 may be performed by photon transmitter104, photon receiver 106, or a combination thereof, withoutcommunicating to a separate device.

In some embodiments, at least some of the tracking functionality may beperformed by other computers, including but not limited to laptopcomputer 112 and/or a mobile computer, such as but not limited to asmartphone or tablet 114. Various embodiments of such computers aredescribed in more detail below in conjunction with mobile computer 200of FIG. 2 and/or network computer 300 of FIG. 3.

Network 102 may be configured to couple network computers with othercomputing devices, including photon transmitter 104, photon receiver106, tracking computer device 110, laptop computer 112, orsmartphone/tablet 114. Network 102 may include various wired and/orwireless technologies for communicating with a remote device, such as,but not limited to, USB cable, Bluetooth®, Wi-Fi®, or the like. In someembodiments, network 102 may be a network configured to couple networkcomputers with other computing devices. In various embodiments,information communicated between devices may include various kinds ofinformation, including, but not limited to, processor-readableinstructions, remote requests, server responses, program modules,applications, raw data, control data, system information (e.g., logfiles), video data, voice data, image data, text data,structured/unstructured data, or the like. In some embodiments, thisinformation may be communicated between devices using one or moretechnologies and/or network protocols.

In some embodiments, such a network may include various wired networks,wireless networks, or various combinations thereof. In variousembodiments, network 102 may be enabled to employ various forms ofcommunication technology, topology, computer-readable media, or thelike, for communicating information from one electronic device toanother. For example, network 102 can include—in addition to theInternet—LANs, WANs, Personal Area Networks (PANs), Campus AreaNetworks, Metropolitan Area Networks (MANs), direct communicationconnections (such as through a universal serial bus (USB) port), or thelike, or various combinations thereof.

In various embodiments, communication links within and/or betweennetworks may include, but are not limited to, twisted wire pair, opticalfibers, open air lasers, coaxial cable, plain old telephone service(POTS), wave guides, acoustics, full or fractional dedicated digitallines (such as T1, T2, T3, or T4), E-carriers, Integrated ServicesDigital Networks (ISDNs), Digital Subscriber Lines (DSLs), wirelesslinks (including satellite links), or other links and/or carriermechanisms known to those skilled in the art. Moreover, communicationlinks may further employ various ones of a variety of digital signalingtechnologies, including without limit, for example, DS-0, DS-1, DS-2,DS-3, DS-4, OC-3, OC-12, OC-48, or the like. In some embodiments, arouter (or other intermediate network device) may act as a link betweenvarious networks including those based on different architectures and/orprotocols to enable information to be transferred from one network toanother. In other embodiments, remote computers and/or other relatedelectronic devices could be connected to a network via a modem andtemporary telephone link. In essence, network 102 may include variouscommunication technologies by which information may travel betweencomputing devices.

Network 102 may, in some embodiments, include various wireless networks,which may be configured to couple various portable network devices,remote computers, wired networks, other wireless networks, or the like.Wireless networks may include various ones of a variety of sub-networksthat may further overlay stand-alone ad-hoc networks, or the like, toprovide an infrastructure-oriented connection for at least clientcomputer (e.g., laptop computer 112 or smart phone or tablet computer114) (or other mobile devices). Such sub-networks may include meshnetworks, Wireless LAN (WLAN) networks, cellular networks, or the like.In at least one of the various embodiments, the system may include morethan one wireless network.

Network 102 may employ a plurality of wired and/or wirelesscommunication protocols and/or technologies. Examples of variousgenerations (e.g., third (3G), fourth (4G), or fifth (5G)) ofcommunication protocols and/or technologies that may be employed by thenetwork may include, but are not limited to, Global System for Mobilecommunication (GSM), General Packet Radio Services (GPRS), Enhanced DataGSM Environment (EDGE), Code Division Multiple Access (CDMA), WidebandCode Division Multiple Access (W-CDMA), Code Division Multiple Access2000 (CDMA2000), High Speed Downlink Packet Access (HSDPA), Long TermEvolution (LTE), Universal Mobile Telecommunications System (UMTS),Evolution-Data Optimized (Ev-DO), Worldwide Interoperability forMicrowave Access (WiMax), time division multiple access (TDMA),Orthogonal frequency-division multiplexing (OFDM), ultra-wide band(UWB), Wireless Application Protocol (WAP), user datagram protocol(UDP), transmission control protocol/Internet protocol (TCP/IP), variousportions of the Open Systems Interconnection (OSI) model protocols,session initiated protocol/real-time transport protocol (SIP/RTP), shortmessage service (SMS), multimedia messaging service (MMS), or variousones of a variety of other communication protocols and/or technologies.In essence, the network may include communication technologies by whichinformation may travel between photon transmitter 104, photon receiver106, and tracking computer device 110, as well as other computingdevices not illustrated.

In various embodiments, at least a portion of network 102 may bearranged as an autonomous system of nodes, links, paths, terminals,gateways, routers, switches, firewalls, load balancers, forwarders,repeaters, optical-electrical converters, or the like, which may beconnected by various communication links. These autonomous systems maybe configured to self-organize based on current operating conditionsand/or rule-based policies, such that the network topology of thenetwork may be modified.

As discussed in detail below, photon transmitter 104 may provide anoptical beacon signal. Accordingly, photon transmitter 104 may include atransmitter (Tx). Photon transmitter 104 may transmit a photon beam ontoa projection surface of target 108. Thus, photon transmitter 104 maytransmit and/or project an image onto the target 108. The image mayinclude a sequential pixilation pattern. The discreet pixels shown onthe surface of target 108 indicate the sequential scanning of pixels ofthe image via sequential scanning performed by photon transmitter 108.Photon receiver (R_(x)) 106 may include an observing system whichreceives the reflect image. As noted, photon receiver 106 may be inmotion relative (as noted by the velocity vector) to the image beingprojected. The relative motion between photon receiver 106 and each ofthe photon transmitter 104 and target 108 may include a relativevelocity in various directions and an arbitrary amplitude. In system100, photon transmitter 104 and the image on the surface are not inrelative motion. Rather, the image is held steady on the surface oftarget 108. However, other embodiments are not so constrained (e.g., thephoton transmitter 104 may be in relative motion to target 108). Theprojected image may be anchored on the surface by compensating for therelative motion between the photon transmitter 104 and the target 108.

Illustrative Mobile Computer

FIG. 2 shows one embodiment of an exemplary mobile computer 200 that mayinclude many more or less components than those exemplary componentsshown. Mobile computer 200 may represent, for example, at least oneembodiment of laptop computer 112, smartphone/tablet 114, and/ortracking computer 110 of system 100 of FIG. 1. Thus, mobile computer 200may include a mobile device (e.g., a smart phone or tablet), astationary/desktop computer, or the like.

Client computer 200 may include processor 202 in communication withmemory 204 via bus 206. Client computer 200 may also include powersupply 208, network interface 210, processor-readable stationary storagedevice 212, processor-readable removable storage device 214,input/output interface 216, camera(s) 218, video interface 220, touchinterface 222, hardware security module (HSM) 224, projector 226,display 228, keypad 230, illuminator 232, audio interface 234, globalpositioning systems (GPS) transceiver 236, open air gesture interface238, temperature interface 240, haptic interface 242, and pointingdevice interface 244. Client computer 200 may optionally communicatewith a base station (not shown), or directly with another computer. Andin one embodiment, although not shown, a gyroscope may be employedwithin client computer 200 for measuring and/or maintaining anorientation of client computer 200.

Power supply 208 may provide power to client computer 200. Arechargeable or non-rechargeable battery may be used to provide power.The power may also be provided by an external power source, such as anAC adapter or a powered docking cradle that supplements and/or rechargesthe battery.

Network interface 210 includes circuitry for coupling client computer200 to one or more networks, and is constructed for use with one or morecommunication protocols and technologies including, but not limited to,protocols and technologies that implement various portions of the OSImodel for mobile communication (GSM), CDMA, time division multipleaccess (TDMA), UDP, TCP/IP, SMS, MMS, GPRS, WAP, UWB, WiMax, SIP/RTP,GPRS, EDGE, WCDMA, LTE, UMTS, OFDM, CDMA2000, EV-DO, HSDPA, or variousones of a variety of other wireless communication protocols. Networkinterface 210 is sometimes known as a transceiver, transceiving device,or network interface card (NIC).

Audio interface 234 may be arranged to produce and receive audio signalssuch as the sound of a human voice. For example, audio interface 234 maybe coupled to a speaker and microphone (not shown) to enabletelecommunication with others and/or generate an audio acknowledgementfor some action. A microphone in audio interface 234 can also be usedfor input to or control of client computer 200, e.g., using voicerecognition, detecting touch based on sound, and the like.

Display 228 may be a liquid crystal display (LCD), gas plasma,electronic ink, light emitting diode (LED), Organic LED (OLED) orvarious other types of light reflective or light transmissive displaysthat can be used with a computer. Display 228 may also include the touchinterface 222 arranged to receive input from an object such as a stylusor a digit from a human hand, and may use resistive, capacitive, surfaceacoustic wave (SAW), infrared, radar, or other technologies to sensetouch and/or gestures.

Projector 226 may be a remote handheld projector or an integratedprojector that is capable of projecting an image on a remote wall orvarious other reflective objects such as a remote screen.

Video interface 220 may be arranged to capture video images, such as astill photo, a video segment, an infrared video, or the like. Forexample, video interface 220 may be coupled to a digital video camera, aweb-camera, or the like. Video interface 220 may comprise a lens, animage sensor, and other electronics. Image sensors may include acomplementary metal-oxide-semiconductor (CMOS) integrated circuit,charge-coupled device (CCD), or various other integrated circuits forsensing light.

Keypad 230 may comprise various input devices arranged to receive inputfrom a user. For example, keypad 230 may include a push button numericdial, or a keyboard. Keypad 230 may also include command buttons thatare associated with selecting and sending images.

Illuminator 232 may provide a status indication and/or provide light.Illuminator 232 may remain active for specific periods of time or inresponse to event messages. For example, if illuminator 232 is active,it may backlight the buttons on keypad 230 and stay on while the clientcomputer is powered. Also, illuminator 232 may backlight these buttonsin various patterns if particular actions are performed, such as dialinganother client computer. Illuminator 232 may also cause light sourcespositioned within a transparent or translucent case of the clientcomputer to illuminate in response to actions.

Further, client computer 200 may also comprise HSM 224 for providingadditional tamper resistant safeguards for generating, storing and/orusing security/cryptographic information such as, keys, digitalcertificates, passwords, passphrases, two-factor authenticationinformation, or the like. In some embodiments, hardware security modulemay be employed to support one or more standard public keyinfrastructures (PKI), and may be employed to generate, manage, and/orstore keys pairs, or the like. In some embodiments, HSM 224 may be astand-alone computer, in other cases, HSM 224 may be arranged as ahardware card that may be added to a client computer.

Client computer 200 may also comprise input/output interface 216 forcommunicating with external peripheral devices or other computers suchas other client computers and network computers. The peripheral devicesmay include an audio headset, virtual reality headsets, display screenglasses, remote speaker system, remote speaker and microphone system,and the like. Input/output interface 216 can utilize one or moretechnologies, such as Universal Serial Bus (USB), Infrared, Wi-Fi™,WiMax, Bluetooth™, and the like.

Input/output interface 216 may also include one or more sensors fordetermining geolocation information (e.g., GPS), monitoring electricalpower conditions (e.g., voltage sensors, current sensors, frequencysensors, and so on), monitoring weather (e.g., thermostats, barometers,anemometers, humidity detectors, precipitation scales, or the like), orthe like. Sensors may be one or more hardware sensors that collectand/or measure data that is external to client computer 200.

Haptic interface 242 may be arranged to provide tactile feedback to auser of the client computer. For example, the haptic interface 242 maybe employed to vibrate client computer 200 in a particular way ifanother user of a computer is calling. Temperature interface 240 may beused to provide a temperature measurement input and/or a temperaturechanging output to a user of client computer 200. Open air gestureinterface 238 may sense physical gestures of a user of client computer200, for example, by using single or stereo video cameras, radar, agyroscopic sensor inside a computer held or worn by the user, or thelike. Camera 218 may be used to track physical eye movements of a userof client computer 200.

GPS transceiver 236 can determine the physical coordinates of clientcomputer 200 on the surface of the Earth, which typically outputs alocation as latitude and longitude values. GPS transceiver 236 can alsoemploy other geo-positioning mechanisms, including, but not limited to,triangulation, assisted GPS (AGPS), Enhanced Observed Time Difference(E-OTD), Cell Identifier (CI), Service Area Identifier (SAI), EnhancedTiming Advance (ETA), Base Station Subsystem (BSS), or the like, tofurther determine the physical location of client computer 200 on thesurface of the Earth. It is understood that under different conditions,GPS transceiver 236 can determine a physical location for clientcomputer 200. In one or more embodiments, however, client computer 200may, through other components, provide other information that may beemployed to determine a physical location of the client computer,including for example, a Media Access Control (MAC) address, IP address,and the like.

Human interface components can be peripheral devices that are physicallyseparate from client computer 200, allowing for remote input and/oroutput to client computer 200. For example, information routed asdescribed here through human interface components such as display 228 orkeypad 230 can instead be routed through network interface 210 toappropriate human interface components located remotely. Examples ofhuman interface peripheral components that may be remote include, butare not limited to, audio devices, pointing devices, keypads, displays,cameras, projectors, and the like. These peripheral components maycommunicate over a Pico Network such as Bluetooth™, Zigbee™ and thelike. One non-limiting example of a client computer with such peripheralhuman interface components is a wearable computer, which might include aremote pico projector along with one or more cameras that remotelycommunicate with a separately located client computer to sense a user'sgestures toward portions of an image projected by the pico projectoronto a reflected surface such as a wall or the user's hand.

Memory 204 may include RAM, ROM, and/or other types of memory. Memory204 illustrates an example of computer-readable storage media (devices)for storage of information such as computer-readable instructions, datastructures, program modules or other data. Memory 204 may store BIOS 246for controlling low-level operation of client computer 200. The memorymay also store operating system 248 for controlling the operation ofclient computer 200. It will be appreciated that this component mayinclude a general-purpose operating system such as a version of UNIX, orLINUX™, or a specialized client computer communication operating systemsuch as Windows Phone™, or the Symbian® operating system. The operatingsystem may include, or interface with a Java virtual machine module thatenables control of hardware components and/or operating systemoperations via Java application programs.

Memory 204 may further include one or more data storage 250, which canbe utilized by client computer 200 to store, among other things,applications 252 and/or other data. For example, data storage 250 mayalso be employed to store information that describes variouscapabilities of client computer 200. In one or more of the variousembodiments, data storage 250 may store tracking information 251. Theinformation 251 may then be provided to another device or computer basedon various ones of a variety of methods, including being sent as part ofa header during a communication, sent upon request, or the like. Datastorage 250 may also be employed to store social networking informationincluding address books, buddy lists, aliases, user profile information,or the like. Data storage 250 may further include program code, data,algorithms, and the like, for use by a processor, such as processor 202to execute and perform actions. In one embodiment, at least some of datastorage 250 might also be stored on another component of client computer200, including, but not limited to, non-transitory processor-readablestationary storage device 212, processor-readable removable storagedevice 214, or even external to the client computer.

Applications 252 may include computer executable instructions which, ifexecuted by client computer 200, transmit, receive, and/or otherwiseprocess instructions and data. Applications 252 may include, forexample, tracking client engine 254, other client engines 256, webbrowser 258, or the like. Client computers may be arranged to exchangecommunications, such as, queries, searches, messages, notificationmessages, event messages, alerts, performance metrics, log data, APIcalls, or the like, combination thereof, with application servers,network file system applications, and/or storage managementapplications.

The web browser engine 226 may be configured to receive and to send webpages, web-based messages, graphics, text, multimedia, and the like. Theclient computer's browser engine 226 may employ virtually variousprogramming languages, including a wireless application protocolmessages (WAP), and the like. In one or more embodiments, the browserengine 258 is enabled to employ Handheld Device Markup Language (HDML),Wireless Markup Language (WML), WMLScript, JavaScript, StandardGeneralized Markup Language (SGML), HyperText Markup Language (HTML),eXtensible Markup Language (XML), HTML5, and the like.

Other examples of application programs include calendars, searchprograms, email client applications, IM applications, SMS applications,Voice Over Internet Protocol (VOIP) applications, contact managers, taskmanagers, transcoders, database programs, word processing programs,security applications, spreadsheet programs, games, search programs, andso forth.

Additionally, in one or more embodiments (not shown in the figures),client computer 200 may include an embedded logic hardware deviceinstead of a CPU, such as, an Application Specific Integrated Circuit(ASIC), Field Programmable Gate Array (FPGA), Programmable Array Logic(PAL), or the like, or combination thereof. The embedded logic hardwaredevice may directly execute its embedded logic to perform actions. Also,in one or more embodiments (not shown in the figures), client computer200 may include a hardware microcontroller instead of a CPU. In one ormore embodiments, the microcontroller may directly execute its ownembedded logic to perform actions and access its own internal memory andits own external Input and Output Interfaces (e.g., hardware pins and/orwireless transceivers) to perform actions, such as System On a Chip(SOC), or the like.

Illustrative Network Computer

FIG. 3 shows one embodiment of an exemplary network computer 300 thatmay be included in an exemplary system implementing one or more of thevarious embodiments. Network computer 300 may include many more or lesscomponents than those shown in FIG. 3. However, the components shown aresufficient to disclose an illustrative embodiment for practicing theseinnovations. Network computer 300 may include a desktop computer, alaptop computer, a server computer, a client computer, and the like.Network computer 300 may represent, for example, one embodiment of oneor more of laptop computer 112, smartphone/tablet 114, and/or trackingcomputer 110 of system 100 of FIG. 1.

As shown in FIG. 3, network computer 300 includes a processor 302 thatmay be in communication with a memory 304 via a bus 306. In someembodiments, processor 302 may be comprised of one or more hardwareprocessors, or one or more processor cores. In some cases, one or moreof the one or more processors may be specialized processors designed toperform one or more specialized actions, such as, those describedherein. Network computer 300 also includes a power supply 308, networkinterface 310, processor-readable stationary storage device 312,processor-readable removable storage device 314, input/output interface316, GPS transceiver 318, display 320, keyboard 322, audio interface324, pointing device interface 326, and HSM 328. Power supply 308provides power to network computer 300.

Network interface 310 includes circuitry for coupling network computer300 to one or more networks, and is constructed for use with one or morecommunication protocols and technologies including, but not limited to,protocols and technologies that implement various portions of the OpenSystems Interconnection model (OSI model), global system for mobilecommunication (GSM), code division multiple access (CDMA), time divisionmultiple access (TDMA), user datagram protocol (UDP), transmissioncontrol protocol/Internet protocol (TCP/IP), Short Message Service(SMS), Multimedia Messaging Service (MIMS), general packet radio service(GPRS), WAP, ultra wide band (UWB), IEEE 802.16 WorldwideInteroperability for Microwave Access (WiMax), Session InitiationProtocol/Real-time Transport Protocol (SIP/RTP), or various ones of avariety of other wired and wireless communication protocols. Networkinterface 310 is sometimes known as a transceiver, transceiving device,or network interface card (NIC). Network computer 300 may optionallycommunicate with a base station (not shown), or directly with anothercomputer.

Audio interface 324 is arranged to produce and receive audio signalssuch as the sound of a human voice. For example, audio interface 324 maybe coupled to a speaker and microphone (not shown) to enabletelecommunication with others and/or generate an audio acknowledgementfor some action. A microphone in audio interface 324 can also be usedfor input to or control of network computer 300, for example, usingvoice recognition.

Display 320 may be a liquid crystal display (LCD), gas plasma,electronic ink, light emitting diode (LED), Organic LED (OLED) orvarious other types of light reflective or light transmissive displaythat can be used with a computer. Display 320 may be a handheldprojector or pico projector capable of projecting an image on a wall orother object.

Network computer 300 may also comprise input/output interface 316 forcommunicating with external devices or computers not shown in FIG. 3.Input/output interface 316 can utilize one or more wired or wirelesscommunication technologies, such as USB™, Firewire™, Wi-Fi™, WiMax,Thunderbolt™, Infrared, Bluetooth™, Zigbee™, serial port, parallel port,and the like.

Also, input/output interface 316 may also include one or more sensorsfor determining geolocation information (e.g., GPS), monitoringelectrical power conditions (e.g., voltage sensors, current sensors,frequency sensors, and so on), monitoring weather (e.g., thermostats,barometers, anemometers, humidity detectors, precipitation scales, orthe like), or the like. Sensors may be one or more hardware sensors thatcollect and/or measure data that is external to network computer 300.Human interface components can be physically separate from networkcomputer 300, allowing for remote input and/or output to networkcomputer 300. For example, information routed as described here throughhuman interface components such as display 320 or keyboard 322 caninstead be routed through the network interface 310 to appropriate humaninterface components located elsewhere on the network. Human interfacecomponents include various components that allow the computer to takeinput from, or send output to, a human user of a computer. Accordingly,pointing devices such as mice, styluses, track balls, or the like, maycommunicate through pointing device interface 326 to receive user input.

GPS transceiver 318 can determine the physical coordinates of networkcomputer 300 on the surface of the Earth, which typically outputs alocation as latitude and longitude values. GPS transceiver 318 can alsoemploy other geo-positioning mechanisms, including, but not limited to,triangulation, assisted GPS (AGPS), Enhanced Observed Time Difference(E-OTD), Cell Identifier (CI), Service Area Identifier (SAI), EnhancedTiming Advance (ETA), Base Station Subsystem (BSS), or the like, tofurther determine the physical location of network computer 300 on thesurface of the Earth. It is understood that under different conditions,GPS transceiver 318 can determine a physical location for networkcomputer 300. In one or more embodiments, however, network computer 300may, through other components, provide other information that may beemployed to determine a physical location of the client computer,including for example, a Media Access Control (MAC) address, IP address,and the like.

Memory 304 may include Random Access Memory (RAM), Read-Only Memory(ROM), and/or other types of memory. Memory 304 illustrates an exampleof computer-readable storage media (devices) for storage of informationsuch as computer-readable instructions, data structures, program modulesor other data. Memory 304 stores a basic input/output system (BIOS) 330for controlling low-level operation of network computer 300. The memoryalso stores an operating system 332 for controlling the operation ofnetwork computer 300. It will be appreciated that this component mayinclude a general-purpose operating system such as a version of UNIX, orLINUX™, or a specialized operating system such as MicrosoftCorporation's Windows® operating system, or the Apple Corporation's IOS®operating system. The operating system may include, or interface with aJava virtual machine module that enables control of hardware componentsand/or operating system operations via Java application programs.Likewise, other runtime environments may be included.

Memory 304 may further include one or more data storage 334, which canbe utilized by network computer 300 to store, among other things,applications 336 and/or other data. For example, data storage 334 mayalso be employed to store information that describes variouscapabilities of network computer 300. In one or more of the variousembodiments, data storage 334 may store tracking information 335. Thetracking information 335 may then be provided to another device orcomputer based on various ones of a variety of methods, including beingsent as part of a header during a communication, sent upon request, orthe like. Data storage 334 may also be employed to store socialnetworking information including address books, buddy lists, aliases,user profile information, or the like. Data storage 334 may furtherinclude program code, data, algorithms, and the like, for use by one ormore processors, such as processor 302 to execute and perform actionssuch as those actions described below. In one embodiment, at least someof data storage 334 might also be stored on another component of networkcomputer 300, including, but not limited to, non-transitory media insidenon-transitory processor-readable stationary storage device 312,processor-readable removable storage device 314, or various othercomputer-readable storage devices within network computer 300, or evenexternal to network computer 300.

Applications 336 may include computer executable instructions which, ifexecuted by network computer 300, transmit, receive, and/or otherwiseprocess messages (e.g., SMS, Multimedia Messaging Service (MMS), InstantMessage (IM), email, and/or other messages), audio, video, and enabletelecommunication with another user of another mobile computer. Otherexamples of application programs include calendars, search programs,email client applications, IM applications, SMS applications, Voice OverInternet Protocol (VOIP) applications, contact managers, task managers,transcoders, database programs, word processing programs, securityapplications, spreadsheet programs, games, search programs, and soforth. Applications 336 may include tracking engine 346 that performsactions further described below. In one or more of the variousembodiments, one or more of the applications may be implemented asmodules and/or components of another application. Further, in one ormore of the various embodiments, applications may be implemented asoperating system extensions, modules, plugins, or the like.

Furthermore, in one or more of the various embodiments, tracking engine346 may be operative in a cloud-based computing environment. In one ormore of the various embodiments, these applications, and others, may beexecuting within virtual machines and/or virtual servers that may bemanaged in a cloud-based based computing environment. In one or more ofthe various embodiments, in this context the applications may flow fromone physical network computer within the cloud-based environment toanother depending on performance and scaling considerationsautomatically managed by the cloud computing environment. Likewise, inone or more of the various embodiments, virtual machines and/or virtualservers dedicated to tracking engine 346 may be provisioned andde-commissioned automatically.

Also, in one or more of the various embodiments, tracking engine 346 orthe like may be located in virtual servers running in a cloud-basedcomputing environment rather than being tied to one or more specificphysical network computers.

Further, network computer 300 may comprise HSM 328 for providingadditional tamper resistant safeguards for generating, storing and/orusing security/cryptographic information such as, keys, digitalcertificates, passwords, passphrases, two-factor authenticationinformation, or the like. In some embodiments, hardware security modulemay be employ to support one or more standard public key infrastructures(PKI), and may be employed to generate, manage, and/or store keys pairs,or the like. In some embodiments, HSM 328 may be a stand-alone networkcomputer, in other cases, HSM 328 may be arranged as a hardware cardthat may be installed in a network computer.

Additionally, in one or more embodiments (not shown in the figures), thenetwork computer may include one or more embedded logic hardware devicesinstead of one or more CPUs, such as, an Application Specific IntegratedCircuits (ASICs), Field Programmable Gate Arrays (FPGAs), ProgrammableArray Logics (PALs), or the like, or combination thereof. The embeddedlogic hardware devices may directly execute embedded logic to performactions. Also, in one or more embodiments (not shown in the figures),the network computer may include one or more hardware microcontrollersinstead of a CPU. In one or more embodiments, the one or moremicrocontrollers may directly execute their own embedded logic toperform actions and access their own internal memory and their ownexternal Input and Output Interfaces (e.g., hardware pins and/orwireless transceivers) to perform actions, such as System On a Chip(SOC), or the like.

Illustrated Sensing Systems

As shown in FIG. 4, an exemplary pixel sequential triangulatedthree-dimensional sensing system 400 may include an exemplary transmitand receive (T_(x)-R_(x)) system 402. In one or more of the variousembodiments, the transmit and receive (T_(x)-R_(x)) system 402 mayinclude a transmit system 404. The transmit system 404 may transmit ascanning beam 406. The transmit system 404 include one or more brightsources 408. For example, the one or more bright sources 408 may includeone or more diode lasers. Light from the one or more bright sources 408may be collimated into the scanning beam 406. The bright sources 408 mayemit the scanning beam 406 toward a beam scanning mechanism 410. Forexample, the beam scanning mechanism 410 may include amicroelectromechanical system (MEMS) mirror, a MEMS-ribbon-activatedphased array, an Optical Phased Array (OPA), a galvanic mirror, or apolygonal rotating mirror. The beam scanning mechanism 410 may spatiallyrotate the scanning beam 406 through a field of view of the transmitsystem 404. For example, the transmit system 404 may project, via thescanning beam 406, one or more pixel sized spots onto one or moreobjects C in the field of view of the transmit system 404.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 402 may include a receive system 412. The receivesystem 412 may include an optical receiver system that contains one ormore position sensors 414 (e.g., a semiconductor sensor) (FIG. 4 showsone row of the one or more sensors 414, yet the one or more sensors 414may include one or more rows) that detect light 416 from the collimatedbeam 406 that reflects off one or more surfaces of one or more objects Cin a field of view of the receiver. An aperture 418 of the receivesystem 412 may capture a fraction of the reflected light 416. A lens oroptical assembly (e.g., at the aperture 418) of the receive system 412may focus captured light into a spot on a surface of the sensor 414. Theposition of the spot may be a geometric function of a scan angle α and arange distance Z between the object C and a base line 420 that extendsbetween the transmit system 404 and the receive system 412.

The transmit and receive (T_(x)-R_(x)) system may employ an offsetdistance D between a portion of the transmit system 404 and a portion ofthe receive system 412. In some of the various embodiments, the offsetdistance D may extend between a scan axis of a scan mirror (e.g., thebeam scanning mechanism 410) of the transmit system 404 and an opticalcenter of receiving optics of the receive system 412 (e.g., a pointwhere a chief ray, such as, for example, the reflected light 416, passesthrough a center of a lens system of the receive system 412). Ahorizontal offset may cause an azimuthal disparity Q (e.g., displacementalong a horizontal offset direction). As illustrated more closely inFIG. 5, the azimuthal disparity Q may also depend on optics (e.g., adistance such as a focal length f between a center of the optics of thereceive system 412 and a surface of the sensor 414 of the receive system412).

FIG. 5 shows an exemplary receive system 500. For example, the receivesystem 500 may be the same as or similar to that of FIG. 4. In some ofthe various embodiments, an angle of beta β may be an angle formed by achief ray 502 with a baseline 504. For example, the angle of beta β maybe formed by the chief ray 502 at a center B of an aperture 506 and thebaseline 504

(FIG. 5 shows a partial view of the baseline 504). In some embodiments,the receive system 502 may include optics that avoid magnification. Forexample, the receive system 500 may cause the chief ray 502 to impingeon a sensor 508 at point I, making the same angle of beta β. In one ormore of the various embodiments, each pixel of the sensor 508 may have acolumn position that is proportional to the angle of beta β. Forexample, a pixel at point I may have a column position (e.g., a columnnumber) that is proportional to the angle of beta β.

Returning to FIG. 4, the transmit and receive (T_(x)-R_(x)) system 402may employ one or more transmit systems 404 and one or more receivesystems 412. In one or more of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 402 may employ more than one transmitsystem 404. For example, one or more of the multiple transmit systems404 may share one or more scan mechanisms 410 (e.g., a shared scanmirror). In other embodiments, one or more of the multiple transmitsystems 404 may each have separate scan mechanisms 410 (e.g., separatescan mirrors). In some of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 402 may employ more than one receive system412 (e.g., multiple receivers).

In one or more of the various embodiments, the transmit system 404 maydetermine (e.g., ex ante) a pointing direction of the scanning beam 406in two dimensions. In some of the various embodiments, the transmitsystem 404 may determine two or more rotational angles of the scanningbeam 406. For example, “at departure,” the transmit system 404 may knowa horizontal pointing angle (e.g., a fast scanning direction) alpha aand a vertical elevation pointing angle epsilon ε (e.g., an angle thatextends from a horizon 422 to the scan beam 406 and/or the reflectedbeam 416).

In one or more of the various embodiments, the receive system 412 maysense an incoming angle of the chief ray 416 in one dimension. In someof the various embodiments, the receive system may determine a receivinghorizontal reflection angle beta β (e.g., an incoming azimuthal angle ofthe portion of the reflected light 416 of the scanning beam 406 that wasreflected by the surface of the object C and captured by the aperture418 of the receive system 412). In some embodiments, the receive system412 may measure an angle at which the chief ray 416 enters the center ofthe aperture 418, optics, or lens surface of the receive system 412. Insome of the various embodiments, the receive system 412 may measure asecond dimension (e.g., elevation according to the vertical deviationangle c away from the horizon 422). In some of the various embodiments,the receive system 412 may measure the second dimension when thescanning beam 406 is pointing upwards (or downwards), above (or below)the horizon 422 of the system 400 (e.g., at angle c). For example, thelight of the scanning beam 406 may be reflected back down (or up)towards the receiver 412 from the same elevation. The light ray 416 mayreflect back to the receiver 412 in the same plane (e.g., a plane formedby an outgoing central ray of the collimated laser beam 406 and thechief ray 416 of a returning ray bundle captured by the receivingoptics).

In one or more of the various embodiments, the sensor 414 of the receivesystem 412 may sense the incoming direction of the reflected light 416in two dimensions (e.g., horizontally (such as, for example, β) andvertically (such as, for example, ε)). In some of the variousembodiments, the receive system 412 may determine the incoming directionof the reflected light 416 in two dimensions by determining both one ormore instantaneous column positions and one or more instantaneous rowpositions for each activated pixel (as explained in further detail belowwith regard to FIG. 6). In some embodiments, the transmit system 406 mayrecord one dimension (e.g., the horizontal pointing direction α) “atdeparture.” The receive system 412 may determine the elevation angle ε.While configuring the receive system 412 to determine the elevationangle ε may require slightly more complexity, this configuration mayprovide other advantages, as explained in further detail below.

FIG. 6 illustrates an exemplary two-dimensional position trackingreceiver 600. For example, the receiver 600 may be the same as orsimilar to one or more of those explained above. In one or more of thevarious embodiments, the receiver 600 may determine a row 602 of a pixelthat detects a captured light spot 604 (e.g., a pixel that detects acenter of the captured light spot 604). In some embodiments, thereceiver 600 may determine an angle of epsilon ε based on the row 602 ofthe pixel. In one or more of the various embodiments, the receiver 600may determine a column 606 of the pixel. In some of the variousembodiments, the receiver 600 may determine an angle of beta β based onthe column 606 of the pixel.

For example, a chief ray 608 may enter an aperture 610 of the receiver600 that directs the chief ray 608 onto a sensor 612 of the receiver600. In one or more of the various embodiments, the receiver 600 maystore an identifier of a particular row 614. The particular row 614 maycontain one or more pixels that may capture the light spot 604 inresponse to the chief ray 608 being perpendicular to a baseline of thereceiver 600 from the perspective of the angle of epsilon ε (e.g., theangle of epsilon ε may be 90 degrees when measured from the baseline).In some of the various embodiments, the receiver 600 may store anidentifier of a particular column. The particular column 616 may containone or more pixels that may capture the light spot 604 in response tothe chief ray 608 being perpendicular to a baseline of the receiver 600from the perspective of the angle of alpha α (e.g., the angle of alpha αmay be 90 degrees when measured from the baseline).

In one or more of the various embodiments, the receiver 600 may measurea first deviation 618 between the row 602 and the particular row 614. Insome embodiments, the receiver 600 may calculate an incoming directionof the chief ray 608 (e.g., in terms of the angle of epsilon 6) based onthe first deviation 618. In some of the various embodiments, thereceiver 600 may measure a second deviation 620 between the column 606and the particular column 616. In some embodiments, the receiver 600 maycalculate an incoming direction of the chief ray 608 (e.g., in terms ofthe angle of beta β) based on the second deviation 620.

Additionally, in this Specification and the corresponding figures,receiving (incoming) azimuthal angles are generally labeled beta,transmitted azimuthal angles are generally labeled alpha, and elevationangles are generally labeled epsilon (in either direction).

Returning to FIG. 4, in one or more of the various embodiments, thesensor 414 of the receive system 412 may be a dual-function sensor. Thedual-function sensor may determine the spot's disparity instantaneously,thereby enabling a real-time calculation of voxel position by creatingsequential voxel-by-voxel trajectories in three-dimensional space. Thedual-function sensor may also capture red, green, and blue (RGB) lightintensity values (e.g., “grey scale” values of recorded primary colors).In some of the various embodiments, the receive system 412 may matchcaptured color values with each voxel position (e.g., three-dimensionalspatial coordinates), which the transmit and receive (T_(x)-R_(x))system 402 calculates from sequentially recorded disparities. In one ormore of the various embodiments, the receive system 412 may utilizediffuse low-intensity ambient light to record hues. In such case, thereceive system 412 may implement longer exposure time periods. In someof the various embodiments, the transmit system 404 may utilize highintensity colored scanning beams 406. In such case, the receive system412 may implement shorter exposure time periods. This dual-functionversion of the receive system 412 may include a more complex sensor asthe sensor 414 (e.g., as explained in further detail below).

In some of the various embodiments, the receive system 412 may implementthe RGB sensor function to employ and capture structured light codes(e.g., as explained in further detail below). The receive system 412 maylocate the object C in three-dimensional space by taking sequentialspatial-temporal measurements of RGB light intensities. In one or moreof the various embodiments, by taking the sequential spatial-temporalmeasurements of RGB light intensities, the receive system 412 maydetermine color and grey scale pixel values and implement these valuesto locate the object C in three-dimensional space by establishing adirect color-coded correlation between each of the incoming light rays416 and matching each of the incoming light rays 416 with one of aplurality of outgoing scan-projected light rays 406. For example, thetransmit and receive (T_(x)-R_(x)) system 402 may implement De Bruijncoding.

In some of the various embodiments, the receive system 412 may implementthe additional RGB sensor function to provide a robust system that mayemploy high-density multi-color structured light code sequences. Forexample, the receive system 412 may employ the high-density multi-colorstructured light code sequences to create an accurate and fine-grainedthree-dimensional surface contour map of the scanned object C.

In some of the various embodiments, the receive system 412 may implementanalog color grey scale ability to detect and compensate for challengingambient light conditions. For example, in response to the receive system412 detecting strong non-uniform lighting, the receive system 412 mayprovide one or more signals to the transmit system 404. In one or moreof the various embodiments, the transmit system 404 may adjust theprojected light 406 to improve a signal-to-noise ratio in response tothe one or more signals from the receive system 412, thereby enhancingrobustness of the system 400. For example, in response to the adjustmentto the projected light 406, the receive system 412 may identify specularreflections and thereby prevent errors.

In one or more of the various embodiments, the receive system 412 maydetect one or more strong non-uniform, highly directional ambient lightsources. In response to detecting the one or more strong non-uniform,highly directional ambient light sources, the receive system 412 mayprovide one or more signals to the transmit system 404. In some of thevarious embodiments, the one or more signals may indicate one or morecharacteristics of the detected ambient light sources.

In response to the one or more signals from the receive system 412, thetransmit system 404 may emit a specifically selected, dynamicallyadjusted mix of R, G & B intensities in or as the scanning beam 406. Forexample, the transmit system 404 may delay emitting the scanning beam406 for a time period sufficient to allow the receive system 412 torecord one or more images. During the time period, the receive system412 may record one or more images in response to light received from theambient light sources. In some of the various embodiments, the sensor414 of the receive system 412 may be a dual-mode sensor. The dual-modesensor of the receive system 412 may employ milliseconds-long exposureduring the time period. The receive system 412 may provide one or moresignals indicative of one or more characteristics of the ambient lightto the transmit system 404. Based on the one or more characteristics ofthe ambient light, the transmit system 404 may dynamically adjust themix and intensities of color sources that feed into the scanning beam406. In some embodiments, the transmit system 404 may strobe thescanning beam 406. In some of the various embodiments, the receivesystem 412 may accurately measure surface contrast (e.g., hue, color,contrast) of the scanned object C by synchronizing measurements by thereceive system 412 with the strobed beam 406. In one or more of thevarious embodiments, the dual mode sensor may enter a scannedsynchronous mode to record observations using the strobed beam 406. Inthis mode, an intense-spot illumination source is synchronizedline-by-line (e.g., the receive system 412 may set an exposure time ofthe sensor 414 to a time period, such as, for example, microseconds, ofexposure that the receive system 412 spatially and temporallysynchronizes with the scan beam 406). In this mode the reflection 416 ofthe synchronized light 406 may have a 500-to-1 (or greater) advantageover the ambient light. Thus, the transmit and receive (T_(x)-R_(x))system 402 may overcome a challenging ambient light condition that couldotherwise inhibit either accurate voxel location tracking orhigh-dynamic range surface color contrast detection. In particular, thetransmit and receive (T_(x)-R_(x)) system 402 may improve dynamic systemperformance.

In one or more of the various embodiments, the transmit system 404 maydetermine elevation of the spot ex-ante. In some of the variousembodiments, transmitting optics (e.g., the beam scanning mechanism 410)may determine the elevation of the spot. For example, a known deflectionangle of the scanning optics, one or more MEMS scan mirrors, or an OPAsystem may indicate the elevation of the spot. Therefore, for each pixelrecorded by a receiving camera (e.g., the sensor 414), a correspondingvoxel position in the field of view can be calculated. As describedpreviously, both three-dimensional surface structures, as well ashigh-resolution image details (contrast functions), can bemotion-captured with fine-grained temporal resolution, resulting inun-blurred high velocity images with fine image details individuallytime-stamped in microseconds and, in some cases, even in nanoseconds.One advantage is that using this more precise intra-frame time (aprecise time interval that the spot was illuminated in a particularsub-frame area) can significantly increase precision of estimations ofan object's position, velocity, and acceleration.

In conventional cameras a typical frame exposure time is measured intens of milliseconds. For example, a 30 FPS camera may have a frameexposure time as long as 33 milliseconds, thereby introducingsignificant motion blur and temporal position ambiguity. This timeambiguity result in velocity ambiguity. For example, this time ambiguitymay result in velocity ambiguity with regard to an observed edge of anobject or structure critical to collision avoidance or with regard topath planning in a fast flying autonomous drone. In contrast, in one ormore of the various embodiments, the transmit and receive (T_(x)-R_(x))system 402 may provide superior collision avoidance. For example, byproviding fast and accurate measurements and calculations, the transmitand receive (T_(x)-R_(x)) system 402 may provide fast and/or agilenavigation. Narrowing down the observation time window, from wholeframes to individual lines and pixels, reduces the uncertainty of anobserved event to a significantly smaller time window (the spotillumination strobe moment). It thereby greatly improves the accuracy ofsuch observations and the accuracy of the predictions based on them.This critically improves real-time trajectory estimations andhigh-speed, low-latency calculations for collision avoidance and enablesjust-in-time, split-second “dodging” of high-speed projectiles.

In one or more of the various embodiments, the highly collimated triggerbeam 406 may have an extent of one pixel. For example, the extent of thetrigger beam 406 may include approximately 1 millionth of the field ofview. In some of the various embodiments, in a high definition (HD)resolution, one pixel scans approximately one million sequentialpositions across the field of view.

In some of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 402 may employ a strobed search light with avoxel-sized three-dimensional trigger beam as the scanning beam 406. Inone or more of the various embodiments, a broad-spectrum illuminationbeam may be arranged to be co-centric with the trigger beam 406. Inother embodiments, the broad-spectrum illumination beam may be arrangedto be in fast pursuit of the trigger beam 406. In some of the variousembodiments, the broad-spectrum illumination beam may have a broaderspot, up to 1,000 times bigger in solid angle than the trigger beam 406.The transmit system 404 may utilize a diffusing effect of a phosphor toconvert, for example, a high intensity, monochromatic coherent lasersource (e.g., 445 nm laser diode) into a broad spectrum, more diffuselight source. By using this more diffuse, incoherent light, and withgreater illumination power, the speed, accuracy, and range of thetransmit and receive (T_(x)-R_(x)) system 402 can be extended.

FIG. 7 shows an exemplary strobed search light (SSL) that an exemplarytransmit and receive (T_(x)-R_(x)) system may employ to scan anexemplary scan area 700. For example, the transmit and receive(T_(x)-R_(x)) system may be the same as or similar to one or more ofthose explained above. In one or more of the various embodiments, thestrobed search light (SSL) may have a voxel-sized trigger beam (TB) 702that scans across a portion of a three-dimensional surface 704 thatfalls within the scan area 700. In some of the various embodiments, thetransmit and receive (T_(x)-R_(x)) system may cause the SSL to have awave front 706 that splashes against the search area 700. Thevoxel-sized trigger beam 702 may impact that three-dimensional surface704 at a voxel-sized point 708 within the scan area 700.

FIG. 8 illustrates an exemplary flying spot style strobed search light(SSL) 800 that an exemplary transmit and receive (T_(x)-R_(x)) systemmay employ to scan an exemplary scan area. For example, the transmit andreceive (T_(x)-R_(x)) system may be the same as or similar to one ormore of those explained above. In one or more of the variousembodiments, the flying spot style strobed search light (SSL) 800 maysurround a narrowly-collimated voxel-sized (e.g., single pixel sized)trigger beam 802.

FIG. 9 shows an exemplary flying spot style strobed search light (SSL)900 that an exemplary transmit and receive (T_(x)-R_(x)) system mayemploy to scan an exemplary scan area. For example, the transmit andreceive (T_(x)-R_(x)) system may be the same as or similar to one ormore of those explained above. In one or more of the variousembodiments, the flying spot style strobed search light (SSL) 900 maytrail a narrowly-collimated voxel-sized trigger beam 902. In some of thevarious embodiments, the trigger beam 902 may ensure that there is asmall, yet sufficient, amount of time (after a target reflects thetrigger beam 902) for newly activated pixels to detect the SSL 900. Forexample, activation of the newly activated pixels may be driven by, andfollow, detection of the reflected light of the trigger beam 902.

Returning to FIG. 4, because the transmit and receive (T_(x)-R_(x))system 402 may cause the spotlight to illuminate the pixels for a shorttime period (e.g., a few microseconds) and because the transmit andreceive (T_(x)-R_(x)) system 402 can correlate an exposure period withone or more observed voxel times, the transmit and receive (T_(x)-R_(x))system 402 may record an RGB image that is both substantially blur-freeand time-constrained to a very narrow time window. For example, atraditional sensor mounted on a car going 20 meters per second (72 km/hror 45 mph) and using a global shutter at 100 frames per second may see apixel blur of 20 cm (approximately two-thirds of a foot) when lookingdown onto pavement, rendering camera resolution meaningless andrendering image reconstruction impossible. The smart spot illuminationof one or more of the various embodiments may reduce this motion blurby, for example, orders of magnitude (e.g., to 20 microns), well belowvarious cameras' resolution limits. For example, in some of the variousembodiments, the transmit and receive (T_(x)-R_(x)) system 402 maynarrow down a conventional 33 millisecond window by a factor of, forexample, 1000. In one or more of the various embodiments, the transmitand receive (T_(x)-R_(x)) system 402 may narrow down the conventional 33millisecond window to achieve an exposure time as short as, for example,33 microseconds.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 402 may, in employing the smart spot illumination,select a transition period that dictates delay of emission of thestrobed search light from time of emission of the trigger beam. In someof the various embodiments, the transition period predetermines theexact time of illumination of the pixels by the strobed search lightrelative to the time of illumination of the pixels by the trigger beam.In some embodiments, because the exact time of illumination by thestrobed search light is predetermined relative to the time ofillumination by the trigger beam, the transmit and receive (T_(x)-R_(x))system 402 may activate pixels at the correct time without fast pixelgating, without additional logic in a camera sensor plane itself (e.g.,a plane of the sensor 414). In contrast, conventional three-dimensionalcapture systems (e.g., those produced by Canesta™, SwissRanger™,Softkinetic™, or others) require complex logic in each pixel and requireadditional transistors that increase minimum pixel size, whichnegatively impacts resolution, size, system power, and cost.

Optionally, in one or more of the various embodiments, individual pixels(or a separate high sensitivity receiver, e.g., a SPAD) of the receivesystem 412 may detect the reflected beam 416. In response to thisdetection, the receive system 412 may trigger successive individualexposure on and off circuits (e.g., as explained in further detailbelow).

In some of the various embodiments, transmit and receive (T_(x)-R_(x))system 402 may employ three-dimensional voxel matched, lasertime-of-flight triggered, RGB flash and corresponding pixel-by-pixelauto-exposure. In one or more of the various embodiments, the triggereddual-function sensor of the transmit and receive (T_(x)-R_(x)) system402 may detect light emitted by one or more intense blue lasers and oneor more phosphors of the transmit system 404. In some of the variousembodiments, the one or more intense blue lasers may emit one or morestrong flashes of intense blue laser light. In some embodiments, the oneor more strong flashes of intense blue laser light may excite one ormore phosphors. For example, in response to the intense blue laserlight, a phosphor may convert most of the blue photon energy of theintense blue laser light into a more diffuse white illumination spot.Such phosphor down-conversion (e.g., from shorter to a longerwavelength) may respond at a certain latency (e.g., a phase shift). Forexample, the phosphor may cause a white broad spectrum light wave frontto follow a narrow band blue laser flash light wave front after a shortdelay. In some of the various embodiments, a phosphor natural lag ofvarious phosphors can be selected to spread out in time variouswavelengths of iridescence response of the phosphors. For example, thevarious phosphors may provide a phosphor lag time that varies based onthe various wavelengths (e.g., from bluish green to red, wherein longerwavelengths may have longer phosphor lag times).

In some of the various embodiments, transmit system 404 may emit a sharpultraviolet (UV) or blue laser pulse induced iridescence wave fronttravelling at the speed of light. In one or more of the variousembodiments, the first light to arrive at the sensor 414 (after thetime-of-flight delay) may include bluish light. The last photons toarrive may provide reddish light. A similar time delay in displays maycause phenomena referred to as “PlainBow.” In one or more of the variousembodiments, fast pixels may detect the front-running blue light (e.g.,originating directly from the blue laser source). The lag in the arrivalof the trailing phosphor light enables a novel trigger function. Forexample, a trigger pixel may first record an origin of light rays thatare first received by that pixel (e.g., a voxel location on a surfacewhere that ray of blue light was reflected (column identifier and/or rowidentifier)) and may also trigger a second integration sensing functionby, for example, unlatching or opening a new path for photodiodegenerated photo current to be captured (e.g., in a floating diffusion).In one or more of the various embodiments, a pulsed laser may directlygenerate a narrow band blue light that is first to arrive at the pixel.In response to the narrow band blue light, the pixel may open a path forsoon-to-arrive light to be spectrally recorded. By employing thistriggered pixel function, pixels may discriminate between having or nothaving light in them. For example, in response to a given pixel havinglight in it, the given pixel may capture color information.

In some of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 402 may employ filter-less color strobe triggeredtime-sequential smart pixels. In one or more of the various embodiments,the receive system 412 may capture triggered pixel photocurrent of anincoming wave front at to. In some of the various embodiments, a row orcolumn sense line may measure the triggered pixel photocurrent. In someof the embodiments, receive system 412 may cause an additional cascadetriggered adjacent pixel to capture later, longer wavelength componentsof the incoming wave front at a slightly delayed exposure moment ti. Byusing the cascaded delay, the receive system 412 may capture various subbands of the spectrum in a pixel color time sequential method without,for example, requiring filters. In one or more of the variousembodiments, the various cascaded photodiodes may include layered orstacked photodiodes with blue sensitivity closer to the surface and redsensitivity deeper below (e.g., four pinned photo diodes (PDD) of FIG.27 might be one above the other).

For example, when there is a small object within range, the receivesystem 412 may automatically capture those RBG pixel values that thetransmit system 404 includes with the scanning beam 406, whilesuppressing at the pixel and time (e.g., nanosecond) levels variousambient/background light. In this manner, the receive system 412 maycreate a strongly selective pixel function, cropping out thethree-dimensional objects in the foreground, recording the briefestpossible moment of exposure. As explained in further detail below, FIG.28 shows an example of a very fast sequentially triggered and/orexternally gated PDD.

Optionally, in one or more of the various embodiments, the pixel servingas a trigger may be a sub-pixel, with selective narrowband filterspreferentially selective to the triggering blue light wavelength, gatingor unlatching a second separate colored (e.g., RGB) pixel.

Both the transmit system 404 and the receive system 412 may include afast real-time position spot location detector. Location detection ofone or two dimensions and optionally a time-of-flight (time of flight)direct time and distance measurement by a sensor and view/cameraperspective estimation is described in PhotonJet U.S. Pat. Nos.8,282,222, 8,430,512, and 8,696,141 assigned to PhotonJet. Such locationdetection is also explained in further detail by later PhotonJet patentapplications that name Gerard Dirk Smits as an inventor.

In some of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 402 may employ assisted stereo. In one or more ofthe various embodiments, the receive system 412 may include twosynchronized row scanning rolling shutter cameras (e.g., as therespective sensors 414 of each receiver) arranged in an epipolar stereoscan arrangement. For example, an offset distance D may separate twoepipolar stereo receivers (e.g., a left receiver and a right receiver).The dual receivers may function as one-dimensional (1D) positiondetectors that may detect azimuthal disparity column values for aprojected spot along a single row preselected by, for example, aimingthe transmitted scan beam 406 along a certain elevation. In some of thevarious embodiments, the transmit system 404 may know an elevation anglec of the certain elevation, for example, ex-ante. At each moment intime, the receive system 412 may read two values, one from each sensor414 of the two epipolar stereo receivers.

From the known elevation, one or more of the transmit system 404 or thereceive system 412 may determine a Y dimension. From one or more of thedual azimuthal alpha or X values, one or more of the transmit system 404or the receive system 412 may determine both X and Z coordinates (orrange). In some of the various embodiments, for every moment, everyposition (e.g., measured to the nanosecond) that the scan beam 406reflects off an object within the dual aperture stereo field of view,the transmit and receive (T_(x)-R_(x)) system 402 may calculate aninstant (X, Y, Z, t) voxel position-time vector. In one or more of thevarious embodiments, the transmit and receive (T_(x)-R_(x)) system 402may determine a voxel position trajectory. In some of the variousembodiments, the transmit and receive (T_(x)-R_(x)) system 402 maydetermine a voxel position trajectory that includes a plurality of theseposition-time vectors. For example, the transmit and receive(T_(x)-R_(x)) system 402 may record and act upon 100 million or more ofthese position-time vectors every second with minimal (e.g.,microsecond) latency

FIG. 10 illustrates an exemplary fast tracking system 1000 that employsthree exemplary one-dimensional real time sensors to track an exemplaryflying spot style SSL. For example, the fast tracking system 1000 may bethe same as or similar to one or more of the transmit and receive(T_(x)-R_(x)) systems explained above. In one or more of the variousembodiments, the fast tracking system 1000 may include a first receiver1002, a second receiver 1004, and, optionally, a third receiver 1006. Insome of the various embodiments, the first receiver 1002 may include afirst sensor 1008. The second receiver 1004 may include a second sensor1010. The third receiver 1006 may include a third sensor 1012. In someembodiments, the first receiver 1002 may include a first aperture 1014that has a center point at position A. The second receiver 1004 mayinclude a second aperture 1016 that has a center point at position B.The third receiver 1006 may include a third aperture 1018 that has acenter point at position E. In some embodiments, the first, second, andthird apertures 1014, 1016, and 1018 may be directionally aligned alonga baseline 1020.

In some embodiments, the two sensors 1008 and 1010 may recordinstantaneous pixel-by-pixel stereo disparity in a fast scanningazimuthal direction, directionally aligned along an optional row selectline 1022. While FIG. 10 shows a transmit system 1024 positioned alongthe baseline 1020, the transmit system 1024 may reside in variouslocations other than on the baseline 1020 (e.g., as also applies to oneor more other various embodiments explained herein).

In one or more of the various embodiments, the optional third sensor1012 may be a one-dimensional sensor. In some of the variousembodiments, the system 1000 may orient the third one-dimensional sensor1012 to measure instantaneous elevation (c) of the flying spot. Forexample, the third one-dimensional sensor 1012 may have an orientationthat is perpendicular to (e.g., rotated 90 degrees with respect to)orientations of the first sensor 1008 and the second sensor 1010 (e.g.,as illustrated in FIG. 10). In some of the various embodiments, thethird one-dimensional sensor 1012 may reside in line with the row selectline 1022.

For example, the apertures 1014, 1016, and 1018 may direct a chief raythat reflects off an object C onto respective ones of the sensors 1008,1010, and 1012. In one or more of the various embodiments, the rowselect line 1022 may activate a pixel row 1026 of the first sensor 1008.In some of the various embodiments, the row select line 1022 mayactivate a pixel row 1028 of the second sensor 1028. A pixel 1030 in theactivated row 1026 may capture the chief ray. The first receiver 1002may determine a column 1032 of the pixel 1030. A pixel 1034 in theactivated row 1028 may capture the chief ray. The second receiver 1004may determine a column 1036 of the pixel 1034. From these measurements,the system 1000 may determine a position of an object C in a firstdimension.

In one or more of the various embodiments, the third receiver 1006 mayactivate a pixel column 1038 of the third sensor 1012 via, for example,a column select line (not shown). A pixel 1040 in the activated column1038 may capture the chief ray. The third receiver 1006 may determine arow 1042 of the pixel 1040. From these measurements, the system 1000 maydetermine the position of the object C in a second dimension. In some ofthe various embodiments, the first and second dimensions may bedimensions in addition to range.

In some of the various embodiments, the system 1000 may include one ormore dual receivers as one or more of the first sensor 1008 or thesecond sensor 1010. In some embodiments, the transmit system 1024 maytrack elevation of a scanning beam 1044. In some of the variousembodiments, the third receiver 1006 may be co-located with the transmitsystem 1024.

FIG. 11 shows an exemplary fast tracking system 1100 that employs threeexemplary one-dimensional real time sensors to track an exemplary flyingspot style SSL. For example, the fast tracking system 1100 may be thesame as or similar to one or more of the transmit and receive(T_(x)-R_(x)) systems explained above. In one or more of the variousembodiments, the fast tracking system 1100 may include a left receiver,a right receiver, and, optionally, an elevation receiver. The leftreceiver may include a first sensor 1102. The right receiver may includea second sensor 1104. The elevation receiver may include a third sensor1106. In some embodiments, the third sensor 1106 may be aone-dimensional sensor. In some of the various embodiments, the system1100 may orient the third sensor to measure instantaneous elevation(e.g., ε) of the flying spot 1108 on an object C. For example, the thirdsensor 1106 may have an orientation that is perpendicular to (e.g.,rotated 90 degrees with respect to) orientations of the first sensor1102 and the second sensor 1104 (e.g., as illustrated in FIG. 11). Insome of the various embodiments, the system 1100 may mount the thirdsensor 1106 to create additional vertical disparity. For example, thesystem 1100 may mount the third sensor 1106 above a vertical position ofthe left receiver, a vertical position of the right receiver, and avertical position of a transmitter 1110 of the system 1100. In some ofthe various embodiments, the left receiver may include a first aperture1112 that directs reflected light onto the first sensor 1102. The rightreceiver may include a second aperture 1114 that directs reflected lightonto the second sensor 1104. A baseline may extend between the firstaperture 1112 and the second aperture 1114. The elevation receiver mayinclude a third aperture 1116. The third aperture 1116 may be at aposition that forms a triangle along a vertical plane with the baseline.In some embodiments, the system 1100 may mount the third sensor 1106 atabove a vertical position of the baseline.

FIG. 12 illustrates an exemplary fast tracking system 1200 that employsthree exemplary one-dimensional real time sensors to track an exemplaryflying spot style SSL. For example, the fast tracking system 1200 may bethe same as or similar to one or more of the transmit and receive(T_(x)-R_(x)) systems explained above. In one or more of the variousembodiments, the fast tracking system 1200 may include a left sensor1202, a right sensor 1204, and, optionally, an elevation sensor 1206. Insome of the various embodiments, the elevation sensor 1206 may be aone-dimensional sensor. In some embodiments, the elevation sensor 1206may be positioned between the left and right sensors 1202 and 1204. Forexample, the elevation sensor 1206 may be positioned on a baseline thatextends between the left sensor 1202 and the right sensor 1204. In oneor more of the various embodiments, the baseline may have a length ofD_(x). The elevation sensor 1206 may be positioned at a distance of halfof D_(x) from each of the left and right sensors 1202 and 1204.

In some of the various embodiments, the system 1200 may provideadditional vertical disparity while illuminating a portion 1208 of anobject C. In some of the various embodiments, a transmit system 1210 maybe mounted at a position that causes a scanning collimated laser beam tobe oriented high above the baseline and looking down at an obstacle(such as, for example, the object C) (e.g., mounted above the obstacle).This may be advantageous in vehicles. In some embodiments, the system1200 may position the transmit system 1210 at a distance of D_(y) abovethe baseline. The system 1200 may position the transmit system 1210vertically above the elevation sensor 1206. For example, the system 1200may provide a maximum horizontal baseline D_(x) (e.g., 120 cm to 200 cmin vehicular headlights). The system 1200 may also provide a significantvertical baseline D_(y) (e.g., 80 cm to 120 cm).

FIG. 13 shows an exemplary receive system 1300 that includes anexemplary stereo pair of receivers. For example, the receive system 1300may be the same as or similar to one or more of those explained above.In one or more of the various embodiments, the receive system 1300 mayinclude a left receiver 1302 and a right receiver 1304. The leftreceiver 1302 may include a first sensor 1306. The right receiver 1304may include a second sensor 1308. The left receiver 1302 may include afirst aperture 1310 that directs light onto the first sensor 1306. Theright receiver 1304 may include a second aperture 1312 that directslight onto the second sensor 1308. A distance D may separate the firstaperture 1310 and the second aperture 1312 from each other. The leftreceiver 1302 and the right receiver 1304 may capture reflections of atransmitted beam 1314.

In some of the various embodiments, the receive system 1300 maydetermine an observable disparity (e.g., an observable distinctionbetween beam angles). For example, as a distance between a target andthe receive system 1300 increases, the observable disparity maydiminish. The receive system 1300 may reduce a potential Z-range errorin response to determining that the observable disparity passes athreshold (e.g., falls below the threshold). For example, the receivesystem 1300 can automatically switch to ranging by time-of-flightmethods (e.g., as explained in further detail below).

In some of the various embodiments, the transmitted beam 1314 mayreflect off an object at a first position 1316 (e.g., at a closedistance). In some embodiments, the transmitted beam 1314 may reflectoff an object at a second position 1318 (e.g., at a medium distance). Insome embodiments, the transmitted beam 1314 may reflect off an object athird position (e.g., at a far distance that resides out of FIG. 13).The left receiver 1302 may receive a light ray 1320 that reflects fromthe object at the first position 1316. The right receiver 1304 mayreceive a light ray 1322 that reflects from the object at the firstposition 1316. For example, a first pixel 1324 of the first sensor 1306may capture the reflected light 1320. A second pixel 1326 of the secondsensor 1308 may capture the reflected light 1322. The left receiver 1302may receive a light ray 1328 that reflects from the object at the secondposition 1318. The right receiver 1304 may receive a light ray 1330 thatreflects from the object at the second position 1318. For example, athird pixel 1332 of the first sensor 1306 may capture the reflectedlight 1328. A fourth pixel 1334 of the second sensor 1308 may capturethe reflected light 1330. The left receiver 1302 may receive a light ray1336 that reflects from the object at the third position. The rightreceiver 1304 may receive a light ray 1338 that reflects from the objectat the third position. For example, a fifth pixel 1340 of the firstsensor 1306 may capture the reflected light 1336. A sixth pixel 1342 ofthe second sensor 1308 may capture the reflected light 1338. In one ormore of the various embodiments, a seventh pixel 1344 of the firstsensor 1306 may capture light reflected from various objects thatapproach a distance of infinity from the first sensor 1306. An eighthpixel 1346 of the second sensor 1308 may capture light reflected fromvarious objects that approach a distance of infinity from the secondsensor 1308.

In one or more of the various embodiments, the system 1300 may have gooddisparity 1348 (e.g., between the first pixel 1324 and the second pixel1326) as occurs when capturing light reflected from near objects. Insome of the various embodiments, the system 1300 may have diminishingdisparity 1350 (e.g., between the third pixel 1332 and the fourth pixel1334) as occurs when capturing light reflected from objects at increaseddistances. For example, when capturing reflections from objectspositioned at far distances (e.g., the third position) the system 1300may have poor disparity 1352 (e.g., between the fifth pixel 1340 and thesixth pixel 1342). In some embodiments, the system 1300 may lack (orpractically lack) disparity 1354 (e.g., between the seventh pixel 1344and the eighth pixel 1346) as occurs when capturing reflections fromobjects positioned at distances that approach infinity.

FIG. 14 illustrates an exemplary modified retro reflective (MRR) target1400 that may enhance tracking for an exemplary transmit and receive(T_(x)-R_(x)) system 1402 (e.g., one or more of those explained above).In some of the various embodiments, the modified retro reflective target1400 may provide a reflection of a scanning laser beam 1404. Themodified retro reflective target 1400 may significantly increase a rangefor assisted stereo triangulation. The modified retro reflective target1400 may significantly increase a range for using stereo receivers forlong range time-of-flight measurements. For example, where triangulationmay lack a sufficient baseline at great distances, the modified retroreflective target 1400 may provide the reflection in a manner thatprovides accurate three-dimensional detection.

In one or more of the various embodiments, the modified retro reflectivetarget 1400 may cause the reflection to have a large signal strength.For example, the modified retro reflective target 1400 may cause thereflection to have a 100× gain as compared to a Lambertian or diffusereflection. In some embodiments, the modified retro reflective target1400 may reflect LIDAR mode pulses to provide reflections that may bestrong enough for a receive system to detect at great ranges.

In some of the various embodiments, the modified retro reflective target1400 may split the reflection into two retro-reflecting ray bundles 1406and 1408. For example, the modified retro reflective target 1400 may aimthe two retro-reflective ray bundles 1406 and 1408at a stereo receiverpair of the receive system (e.g., each of the two retro-reflective raybundles 1406 and 1408may be aimed at a respective receiver 1412 and 1414of the stereo receiver pair). In some embodiments, the modified retroreflective target 1400 may aim the two retro-reflecting ray bundles 1406and 1408at the stereo receiver pair with an epipolar (e.g., horizontal)diversion angle of ≈3α (e.g., as explained in further detail below). Forexample, the retro reflective target 1400 may split the reflection intoa narrow spread angle reflection beam pair (e.g., with a beam spreadangle that can be less than 1 degree). In one or more of the variousembodiments, the modified retro reflective target 1400 may be decaladhesive MRR targets (e.g., attached at perimeters of vehicles toprovide accurate perception of them by machine vision at greatdistances). By implementing the epipolar (e.g., horizontal) precise anddeliberate 3α spreading by the MRR 1400, the modified retro reflectivetarget 1400 may cause conventional retro-reflecting and otherconspicuous surfaces (e.g., those commonly used in license plates, rearsignals, highway signs, and “cat eye” lane markers) may appear lessbright to the stereo receiver pair of the receive system (e.g., in thehead light assembly). By reflecting light back precisely on target (andoptionally while the transmit system 1410 pulses the light with knownintervals), the modified retro reflective target 1400 may permit thetransmit system 1410 to reduce a required magnitude of transmittedlight. Also by reflecting light back precisely on target (and optionallywhile the transmit system 1410 pulses the light with known intervals),the modified retro reflective target 1400 may permit the receive systemof a vehicle's machine vision guidance system to instantly (orpractically instantly) recognized its own signal.

FIG. 15 shows an exemplary cubic retro-reflective target 1500 that maysplit and reflect an exemplary scanning beam. For example, the cubicretro-reflective target 1500 may be the same as or similar to themodified retro reflective target of FIG. 16. The cubic retro-reflectivetarget may be a component of one or more of the above-explained systems.In one or more of the various embodiments, the cubic retro-reflectivetarget 1500 may have a modified facet 1502 that provides a horizontalreflection split. In some of the various embodiments, the modified facetmay have a small deviation angle α. For example, the angle α mayrepresent an angle between the modified facet 1502 and a plane that isorthogonal to one or more of each other facet of the cubicretro-reflective target 1500 (e.g., an angle between the modified facet1502 and one or more of each other facet is 90 −/+ the small deviationangle α). As explained above with regard to FIG. 14, the cubicretro-reflective target 1500 may provide an angular separation betweentwo reflected beams of three times the deviation angle α.

In some of the various embodiments, when stereo receivers of a transmitand receive (T_(x)-R_(x)) system each simultaneously receive equalamounts of light, the system may increase a degree of certainty that thereceived light is from a transmitter of the system, as opposed tospurious ambient rays (which are likely to arrive at the stereoreceivers at different times or different magnitudes).

FIG. 16 illustrates an exemplary retro-reflective target 1600 that maysplit and reflect an exemplary scanning beam 1602 into two exemplaryseparate beams 1604 and 1606 toward an exemplary vehicle 1608. Forexample, the retro-reflective target 1600 may be the same as or similarto one or more of those explained above. In one or more of the variousembodiments, the vehicle 1608 may include a transmit and receive(T_(x)-R_(x)) system. For example, the transmit and receive(T_(x)-R_(x)) system may be the same as or similar to one or more ofthose explained above. In some of the various embodiments, the transmitand receive (T_(x)-R_(x)) system may include a transmit system 1610. Insome embodiments, the transmit and receive (T_(x)-R_(x)) system mayinclude a first receive system 1612 and a second receive system 1614. Inone or more of the various embodiments, the transmit system 1610 mayemit the transmitting scanning beam 1602.

In one or more of the various embodiments, the vehicle 1608 may be anultra-small vehicle (USV). In some of the various embodiments, thevehicle 1608 may keep accurate range of the retro-reflective target 1600with stereo receivers (e.g., the receive systems 1612 and 1614) builtinto headlight assemblies of the vehicle 1608. The retro-reflectivetarget, for example, may increase an ability of the receive systems 1612and 1614 to detect the reflections 1604 and 1606 of the transmitted beam1602, thereby permitting the transmit system 1610 to emit a very weaksignal (e.g., laser safe) while the receive systems 1612 and 1614provide the accurate range detection. In some of the variousembodiments, due to the very narrow reflection, the transmit system 1610and the retro-reflective target 1600 may avoid detection of reflectionby receivers of other vehicles.

In one or more of the various embodiments, a system of highlydirection-selective reflective targets placed on USVs such as describedhere (for example, especially when adopted as a standard for autonomousUSVs in high-speed lanes) may permit very close and safe platooning athigh speeds. In some of the various embodiments, the high degree ofdirectional selectivity of such targets may minimize potentialinterference effects otherwise caused by direct illuminations or LIDARsystems of vehicles in opposite or adjacent lanes. In some embodiments,when a system simultaneously detects both a left reflection and a rightreflection, the system may associate a resulting calculated distance orposition estimate with a high confidence level because light from anysource other than a retro-reflection of that system's own scanningillumination (for example, bi-directional retro-reflection withapproximate separation angle of 3 alpha as shown in FIG. 15) is unlikelyto result in simultaneously received left and right reflections (forexample, a spurious reflection of another light source is unlikely tosimultaneously arrive in both a left and a right receiver of thesystem).

FIG. 17 shows an exemplary leading vehicle 1700 and an exemplaryfollowing vehicle 1702. For example, one or more of the leading vehicle1700 or the following vehicle 1702 may be the same as or similar to oneor more of those explained above. In one or more of the variousembodiments, the following vehicle 1702 may include a transmit andreceive (T_(x)-R_(x)) system 1704 that is the same as or similar to oneor more of those explained above. In some of the various embodiments,the transmit and receive (T_(x)-R_(x)) system 1704 may include atransmit system 1706. In some embodiments, the transmit and receive(T_(x)-R_(x)) system 1704 may include a first receive system 1708 and asecond receive system 1710. For example, one or more of the firstreceive system 1708 or the second receive system 1710 may be positionedin headlights of the following vehicle 1702.

In one or more of the various embodiments, the transmit system 1706 mayemit a scanning beam in a first direction 1712. In some of the variousembodiments, the transmit system 1706 may emit the scanning beam in asecond direction 1714. In one or more of the various embodiments, theleading vehicle 1700 may include one or more retro-reflectors 1716 and1718. The one or more retro-reflectors 1716 and 1718 may be the same asor similar to one or more of those explained above. For example, the oneor more retro-reflectors 1716 or 1718 may be positioned in one or moretaillights of the leading vehicle 1700. In some of the variousembodiments, the retro-reflector 1718 may split and reflect the scanningbeam at the second position 1714 as first and second reflected beams1720 and 1722 that the first and second receive systems 1708 and 1710capture. In some embodiments, the retro-reflector 1718 may split andreflect the scanning beam at the first position 1712 as third and fourthreflected beams 1724 and 1726 that the first and second receive systems1708 and 1710 capture.

In one or more of the various embodiments, the leading vehicle 1700 andthe following vehicle 1702 may convoy and platoon in a precise line atgreat speed. In some of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 1704 of the following vehicle 1702 maydetect and track the retro-reflectors 1716 and 1718 of the leadingvehicle 1700. In some embodiments, the transmit and receive(T_(x)-R_(x)) system 1704 may determine an orientation of the leadingvehicle 1700. In some embodiments, the transmit and receive(T_(x)-R_(x)) system 1704 may determine an orientation of the followingvehicle 1702. In some embodiments, the transmit and receive(T_(x)-R_(x)) system 1704 may determine one or more of the orientationof the leading vehicle 1700 or the orientation of the following vehicle1702 relative to the other vehicle. For example, the transmit andreceive (T_(x)-R_(x)) system 1704 may mark precise locations of aplurality of retro-reflectors of the leading vehicle 1700 (e.g., 3 or 4targets on the rear of the leading vehicle 1700). The transmit andreceive (T_(x)-R_(x)) system 1704 may fully, precisely, andinstantaneously estimate distances or positions of these targets in sixdegrees of freedom (DOF) via one or more of time-of-flight,triangulation, or photogrammetry methods (for example, estimatingrelative position or velocity between two vehicles). For example, thetransmit and receive (T_(x)-R_(x)) system 1704 may locate each of thethree or four targets on the rear of the leading vehicle 1700 in six DOFafter several repeated measurements (for example, providing informationfor an automated collision avoidance system).

FIG. 18 illustrates an exemplary logical flow diagram for exemplaryprocess 1800 for dynamically switching from triangulation to time offlight. In one or more of the various embodiments, process 1800 maycombine triangulation and time of flight. A transmit and receive(T_(x)-R_(x)) system may employ process 1800. For example, the transmitand receive (T_(x)-R_(x)) system may be the same as or similar to one ormore of those explained above.

After a start block, at block 1802, a transmit system of the transmitand receive (T_(x)-R_(x)) system may send out one or more beams to asurface of an object at position C.

At block 1804, the transmit and receive (T_(x)-R_(x)) system maydetermine whether a receive system of the transmit and receive(T_(x)-R_(x)) system receives one or more reflections of the one or morebeams from the surface at position C (e.g., if position C is within acertain range of the receive system). For example, one or more receiversof the receive system may detect the reflection. In one or more of thevarious embodiments, the transmit and receive system may determinewhether a first receiver of the one or more receivers of the receivesystem receives one or more reflections of the one or more beams fromthe surface at position C. In response to a determination that the firstreceiver fails to receive one or more reflections of the one or morebeams from the surface at position C, the transmit and receive(T_(x)-R_(x)) system may proceed to block 1810. In response to adetermination that the first receiver receives one or more reflectionsof the one or more beams from the surface at position C, the transmitand receive (T_(x)-R_(x)) system may record a time t at which the firstreceiver receives one or more reflections of the one or more beams fromthe surface at position C and may continue to block 1806.

Additionally, in one or more embodiments, when the distance is too greatfor a continuous light beam, the one or more light beams may be pulsedas discussed below for block 1816. Furthermore, in one or moreembodiments, for determining triangulation, e.g., assisted stereo style,both receivers could simultaneously detect the reflected block beam asdiscussed below for block 1808.

At block 1806, the transmit and receive (T_(x)-R_(x)) system maydetermine whether a second receiver of the one or more receivers of thereceive system receives one or more reflections of the one or more beamsfrom the surface at position C. In response to a determination that thesecond receiver fails to receive one or more reflections of the one ormore beams from the surface at position C, the transmit and receive(T_(x)-R_(x)) system may proceed to block 1810. In response to adetermination that the second receiver receives one or more reflectionsof the one or more beams from the surface at position C, the transmitand receive (T_(x)-R_(x)) system may record a time t at which the secondreceiver receives one or more reflections of the one or more beams fromthe surface at position C and may continue to block 1808. In one or moreof the various embodiments, the transmit and receive (T_(x)-R_(x))system may execute blocks 1804 and 1806 in parallel. For example, thetransmit and receive (T_(x)-R_(x)) system may proceed to block 1810 inresponse to one or more of the first or second receivers failing toreceive one or more reflections of the one or more beams from thesurface at position C.

At block 1808, the transmit and receive (T_(x)-R_(x)) system may comparethe time t recorded for the first receiver to the time t recorded forthe second receiver. In one or more of the various embodiments, thetransmit and receive (T_(x)-R_(x)) system may compare these times t todetermine whether these times t indicate that the first and secondreceivers received one or more reflections of the same one or morebeams. For example, the transmit and receive (T_(x)-R_(x)) system maycalculate a difference between the time t recorded for the firstreceiver and the time t recorded for the second receiver. In some of thevarious embodiments, the transmit and receive (T_(x)-R_(x)) system maydetermine whether the difference falls within a predefined threshold. Insome embodiments, the transmit and receive (T_(x)-R_(x)) system mayselect the predefined threshold based on a magnitude of one or more ofthese times t (e.g., the transmit and receive (T_(x)-R_(x)) system mayselect a low predefined threshold in response to one or more of thesetimes t having low magnitude or may select a high predefined thresholdin response to one or more of these times t having high magnitude). Forexample, the transmit and receive (T_(x)-R_(x)) system may select thepredefined threshold based on contents of a lookup table. In response tothe difference falling within the predefined threshold, the transmit andreceive (T_(x)-R_(x)) system may determine that these times t indicatethat the first and second receivers received one or more reflections ofthe same one or more beams. In response to a determination that thesetimes t fail to indicate that the first and second receivers receivedone or more reflections of the same one or more beams, the transmit andreceive (T_(x)-R_(x)) system may proceed to block 1810. In response to adetermination that these times t indicate that the first and secondreceivers received one or more reflections of the same one or morebeams, the transmit and receive (T_(x)-R_(x)) system may continue toblock 1812.

At block 1810, the transmit and receive (T_(x)-R_(x)) system may comparethe number of iterations of block 1802 to a threshold value. Forexample, the transmit and receive (T_(x)-R_(x)) system may increment acounter each time the transmit and receive (T_(x)-R_(x)) system executesblock 1802. In one or more of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system may reset the number of iterationsresponsive to execution of one or more of blocks 1812-1822. In responseto a determination that the number of iterations of block 1802 exceedsthe threshold value, the transmit and receive (T_(x)-R_(x)) system mayproceed to block 1818. For example, in the case where position C is toofar from the transmit and receive (T_(x)-R_(x)) system for bothreceivers to simultaneously receive reflections of a continuous beamthat may adhere to exposure limits to human eye safety such as, forexample, defined by International Electrotechnical Commission (IEC)Document Nos. 60825 or 62471 or American National Standards Institute(ANSI) Z136 (or in the case where there is no object at position C), thetransmit and receive (T_(x)-R_(x)) system may pulse a scanning beam at ahigher intensity that may also adhere to exposure limits to human eyesafety at block 1818. In response to a determination that the number ofiterations of block 1802 fails to exceed the threshold value, thetransmit and receive (T_(x)-R_(x)) system may return to and repeat block1802.

At block 1812, the transmit and receive (T_(x)-R_(x)) system maycalculate a value of a disparity Δx from a stereo pair of pixelpositions (e.g., positions of pixels of a stereo pair of receivers ofthe receive (R_(x)) system as illustrated, for example, in FIG. 13). Insome of the various embodiments, the transmit and receive (T_(x)-R_(x))system may have a large distance measurement error in response to asmall disparity when the position C is far from the transmit and receive(T_(x)-R_(x)) system (e.g., as shown in FIG. 13).

At block 1814, the transmit and receive (T_(x)-R_(x)) system maydetermine whether a value of the disparity Δx falls below a certainminimum value. In response to the disparity value Δx failing to fallbelow the certain minimum value, Δxmin, the transmit and receive(T_(x)-R_(x)) system may continue to block 1816. In response to thedisparity value Δx falling below the certain minimum value, Δxmin, thetransmit and receive (T_(x)-R_(x)) system may continue to block 1818.

At block 1816, the transmit and receive (T_(x)-R_(x)) system may attempttriangulation. In one or more of the various embodiments, the transmitand receive (T_(x)-R_(x)) system may compute a range (e.g., Cz) for adetermined Cx. In some of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system may compute the range by a lookup table. Insome embodiments, the transmit and receive (T_(x)-R_(x)) system maycalculate the range via triangulation. The transmit and receive(T_(x)-R_(x)) system may provide the range to an external system (e.g.,alert system or autopilot system) for further analysis or control overone or more systems (e.g., alert system or autopilot system). Inresponse to one or more of determining or providing the range, thetransmit and receive (T_(x)-R_(x)) system may return to block 1802.

At block 1818, the transmit system may pulse (or otherwise modulate) ascanning beam at low duty cycle. In one or more of the variousembodiments, the transmit system may emit short sharp intense bursts ofphotons. Alternatively, the transmit and receive (T_(x)-R_(x)) systemmay switch to employ another form of ToF ranging. In some embodiments,the transmit system may employ fast amplitude or frequency modulation oflight pulses. In one or more of the various embodiments, in response tothe transmit and receive (T_(x)-R_(x)) system proceeding to block 1818from block 1810, the transmit and receive (T_(x)-R_(x)) system mayemploy different sensors such as, for example, LIDAR or radar to detectthe distance to the object at position C. For example, as discussedabove, the distance to position C from the transmit and receive(T_(x)-R_(x)) system may be too far from the transmit and receive(T_(x)-R_(x)) system to perform triangulation based on a safe-amplitudecontinuous beam.

At block 1818, the receive system may detect these bursts. The transmitand receive (T_(x)-R_(x)) system may determine one or more observed timeof flights of these bursts.

At block 1820, the transmit and receive (T_(x)-R_(x)) system maycalculate a range value Z from the observed time of flight. The transmitand receive (T_(x)-R_(x)) system may provide the range to an externalsystem (e.g., alert system or autopilot system) for further analysis orcontrol over one or more systems (e.g., alert system or autopilotsystem). In response to one or more of determining or providing therange, the transmit and receive (T_(x)-R_(x)) system may return to block1802.

In one or more of the various embodiments, the calculated range value Zfrom the observed time of flight may have a more accurate value ascompared to a value that the transmit and receive (T_(x)-R_(x)) systemwould have calculated via triangulation (e.g., when the disparity valueΔAx falls below the certain minimum value, Δx_(min)). For example, thetransmit and receive (T_(x)-R_(x)) system may include a GHz clock timer.With the GHz clock timer, the transmit and receive (T_(x)-R_(x)) systemmay measure, for example, a 200 ft range with an accuracy of ½ foot(e.g., 0.25% error range), which is likely to be more accurate than acorresponding value calculated from stereo disparity.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system may provide the measured range with accuracy as aone-shot value available immediately (or practically immediately) afterreceiving a pulse. In some of the various embodiments, more than onereceiver may simultaneously (or practically simultaneously) receive thepulse. In some embodiments, the transmit and receive (T_(x)-R_(x))system may compare signals and times of flights of the multiplereceivers for greater accuracy. In some of the various embodiments, thereceivers may receive a modulated signal with slightly offset phasedelays. In this case, the transmit and receive (T_(x)-R_(x)) system mayuse the phase difference to add precision to the ranging calculation.

Unlike calculation error of the triangulation measurements by thetransmit and receive (T_(x)-R_(x)) system, calculation error of thetime-of-flight measurements by the transmit and receive (T_(x)-R_(x))system depends on having a sufficiently large reflected signal within ameasurement time interval (e.g., one nanosecond). In one or more of thevarious embodiments, the transmit and receive (T_(x)-R_(x)) system mayprevent the calculation error of the time-of-flight measurements fromdepending on resolution of a sensor or on a baseline separation of thestereo receivers. In some of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system may maintain XY resolution intime-of-flight mode. In some embodiments, XY resolution of the transmitand receive (T_(x)-R_(x)) system may be determined by one or more of thereceivers, a pointing accuracy of a scan system of the transmit system,or beam tip quality.

Further, to precisely align an image of a laser illuminated spot along arow in a sensor at all distances (ranges), it may be necessary for ascan beam to have a planar fast scan trajectory (e.g., the scan beamrotates in only one direction at a time). A gimbaled two-dimensionalMEMS mirror (e.g., a MEMS mirror produced by Microvision™) angled atsignificant elevations above or below a horizon (ε>>0 degrees or ε<<0degrees) may trace a curved trajectory in space (e.g., as a projectionthe MEMS mirror may manifest a “pin-cushion” distortion). This type ofscan distortion may be strong at significant elevations above or belowthe horizon (e.g., when ε≠0, slow axis rotation—the gimbal framerotation—may cause fast axis of rotation—inner axis on the gimbal—to betilted significantly out of plane). Second fast axis of rotation may benon-orthogonal to an incoming collimated laser projection beam,introducing an additional undesirable compound angle component. Opticalrectification by a receiver of such a curved trajectory (e.g., makingthe curved trajectory appear flat) may be unavailable in a triangulationsystem because a perspective of the receiver is at a different anglefrom a transmitter in the triangulation system. Therefore, precisealignment (e.g., precise line-by-line correspondence) may be unavailablefor fast resonant scanning at top and bottom areas of a receiver sensorin a conventional scanning system. In contrast, in a slow system whereboth axes of scanning can be controlled for every scan position, thispincushion distortion can be compensated for by adjusting an angleduring the scan. This may, however, severely limit scanning frequency toa maximum speed that the slower axis adjustment can achieve. This may bedifficult or impossible to realize in fast resonant scanning systems.Furthermore, this may require significantly more power to drive a mirrorsimultaneously in both directions in a forced non-resonant motion. Incontrast, in some of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 402 may prevent pin-cushion and barrel distortionthat, for example, a MEMS scanning system of the transmit system 404 mayotherwise induce.

FIG. 19 shows an exemplary beam scanning mechanism 1900. For example,the beam scanning mechanism 1900 may be the same as or similar to one ormore of those explained above. The beam scanning mechanism 1900 mayinclude a dual MEMS structure 1902 that combines a first one-dimensionalgimbaled scan mirror 1904 and a second two-dimensional gimbaled scanmirror 1906. In one or more of the various embodiments, a common outerframe may hold the first and second mirrors 1904 and 1906 at tworespective foci 1908 and 1910. In some of the various embodiments, thecommon outer frame may be rotated by 2ε.

In some of the various embodiments, the first mirror 1904 maymechanically rotate in a first direction by c and optically rotate anincoming light 1912 (e.g., from a light source) in the first direction(e.g., as shown by a deflection (2ε)) as a first reflected light 1914toward a relay surface 1916 (e.g., an elliptical relay surface). Thisoptical rotation of the incoming light 1912 may be referred to as“pre-rotation” of the incoming light 1912. The relay surface 1916 mayreflect the first reflected light 1914 as second reflected light 1918toward the second mirror 1906. The second mirror 1906 may reflect thesecond reflected light 1918 as a scanning beam 1920 that the beamscanning mechanism 1900 may scan over a field of view.

In one or more of the various embodiments, as shown in the side view1922 of the beam scanning mechanism 1900 taken along a slow scanningaxis 1924 (e.g., axis of angle epsilon ε, elevation, or Y), the firstreflected light 1914, second reflected light 1918, and scanning beam1920 may be, after the pre-rotation of 2ε, in a plane that is orthogonalto a fast scan axis 1926 (e.g., axis of angle alpha a, horizontal, or X)regardless of rotation about the slow axis 1924. For example, after thereflection on the relay surface 1916, the second reflected light 1918may travel along a path that is orthogonal to the fast scanning axis1926. In some of the various embodiments, this orthogonality mayeliminate any compound angle that may otherwise cause a “pincushion”effect as the second mirror 1906 rotates about the slow scan axis 1924.In some of the various embodiments, the first mirror 1904 may have adeflection of ½ of deflection of the second mirror 1906.

In one or more of the various embodiments, a receive system may includeother relay mechanisms and/or other mirror configurations whileachieving the same principle, ensuring that the direction of the secondreflected beam 1906 remains orthogonal to the fast scan axis 1926 of thesecond mirror 1906 regardless of the slow axis rotation. In some of thevarious embodiments, the beam scanning mechanism 1900 may achieve a“flat” fast scan line. For example, the beam scanning mechanism 1900 mayprovide a perfect or nearly perfect epipolar alignment between variousscan lines in frame and various rows in a sensor. In one or more of thevarious embodiments, the beam scanning mechanism 1900 may compensate fordistortions that sensor optics introduce (e.g., pin cushion and/orbarrel). Such advantages may come at a cost of complicating the beamscanning mechanism 1900.

FIG. 20 shows an exemplary sensor grid 2000. In one or more of thevarious embodiments, a receive system may include the sensor grid 2000as a component of one or more sensors. For example, the receive systemmay be the same as or similar to one or more of those explained above.In some of the various embodiments, the sensor grid 2000 may adjust foroptical distortions by one or more lenses. In one or more of the variousembodiments, the sensor grid 2000 may adjust curve of rows in a sensorplane and, in some embodiments, optionally one or more of size, shape,or aspect ratios of one or more pixels to compensate for opticaldistortions.

In one or more of the various embodiments, the sensor grid 2000 may haveone or more of pixel or row geometries that match fast line scantrajectories. For example, the one or more lenses may be part of a fixedlens system. In one or more of the various embodiments, optics of theone or more lenses may cause pincushion distortion at a sensor plane(e.g., magnification of the one or more lenses may be proportional to aradial distance from an optical center), thereby making corners appear“stretched away”. In some of the various embodiments, the sensor grid2000 may adjust a pixel grid to match the distortion. In someembodiments, the sensor grid 2000 may adjust the pixel grid to matchrows in the sensor with straight scan lines to exploit one or more ofthe epipolar arrangements described above. In one or more of the variousembodiments, line i and line i+1 may be curved. In some of the variousembodiments, the sensor grid 2000 may adjust pixel geometries to matchthe distortion. In some embodiments, the sensor grid 2000 may adjust thepixel geometries to match the straight scan lines to exploit the one ormore of the epipolar arrangements described above. In one or more of thevarious embodiments, one or more of pixel orientations, sizes, or shapesmay change along the scan line direction.

FIG. 21 illustrates an exemplary scanning system 2100. In one or more ofthe various embodiments, the scanning system 2100 may include a transmitand receive (T_(x)-R_(x)) system 2102. For example, the scanning system2100 may be the same as or similar to one or more of those explainedabove (e.g., the transmit and receive (T_(x)-R_(x)) system 2102 may bethe same as or similar to one or more of those explained above). In oneor more of the various embodiments, the scanning system 2100 may includea multi-focus camera array. In some of the various embodiments, themulti-focus camera array may provide overlapping fields of view. Forexample, the scanning system 2100 may employ a wide field of view 2104or a narrow field of view 2106. In one or more of the variousembodiments, the scanning system 2100 may be part of or mounted to avehicle 2108. The wide field of view 2104 may cover one or more portionsof a first lane 2110 and a second lane 2112. The narrow field of view2106 may mainly cover one or more portions of the second lane 2112.

In one or more of the various embodiments, the multi-focus camera arraymay include a sensor that has a given resolution (e.g., 10 Mpixels, 5000columns with 2000 rows). In some of the various embodiments, themulti-focus camera array may be arranged to selectively support the widefield of view 2104 or the narrow field of view 2106. For example, themulti-focus camera may support a wide field of view 2104 (e.g., a fieldof view that has a horizontal span of 83 degrees) distributed over agiven number of columns (e.g., 5000 columns). While supporting the widefield of view 2104, the multi-focus camera may have a giventriangulation capability (e.g., 60 columns per degree, whichapproximates HD or 20/20 human vision). In some of the variousembodiments, if the same sensor is combined with a longer telephoto lenssystem, it may span fewer degrees (e.g., 20 degrees), yielding a higherresolution (e.g., 4× higher resolution of 250 pixels per degree, whichapproximates 4× 20/20 human vision). In some embodiments, with enoughlight, the sensor may resolve about 4× more detail on critical X and Zdimensions. In one or more of the various embodiments, for a givensensor size (e.g., pixels and resolution) various levels of lightgathering can be accomplished.

FIG. 22 shows an exemplary scanning system 2200 with an exemplarymulti-focus camera array 2202 that has an exemplary selected distance2204 between an exemplary sensor 2206 and an exemplary aperture 2208.For example, the scanning system 2200 may be the same as or similar toone or more of those explained above. The multi-focus camera array 2202may selectively employ the selected distance 2204 to provide a wideangle field of view 2210.

FIG. 23 illustrates an exemplary scanning system 2300 with an exemplarymulti-focus camera array 2302 that has an exemplary selected distance2304 between an exemplary sensor 2306 and an exemplary aperture 2308.For example, the scanning system 2300 may be the same as or similar toone or more of those explained above. The multi-focus camera array 2302may selectively employ the selected distance 2304 to provide a narrowangle field of view 2310. Contrasting FIGS. 22 and 23 demonstratestradeoffs between a wide angle field of view and a narrow angle field ofview. In one or more of the various embodiments, one or more of theabove explained receive systems may include a telescopic optical systemthat may have a larger aperture (e.g., lens diameter) to gather morelight and extend the system's range.

Returning to FIG. 4, the transmit and receive (T_(x)-R_(x)) system 402may provide robust high-speed 4K resolution three-dimensional videomotion capture. With a 1 GHz clock, for example, the receive system 412may record up to 1 billion distinct pixel pair disparities. In one ormore of the various embodiments, the receive system 412 may observe upto 1 billion voxels per second. For example, the receive system 412 maysuccessively observe 10,000 voxels per line and up to 100,000lines/second. In some of the various embodiments, the receive system 412may capture with 4K resolution, three-dimensional video at up to 50 fps.For example, the receive system 412 may read a fine-grained 4Kresolution three-dimensional surface contour without motion artifacts,even in strong ambient light conditions.

In one or more of the various embodiments, the receive system 412 mayimplement one or more of higher frame speeds or higher line scanresolution. In some of the various embodiments, the receive system 412may avoid speeding up the scan mechanism. For example, the receivesystem 412 may implement a 50 kHz mirror to record 100,000 lines persecond with a single flying spot or 2,000 lines at up to 50 FPS. In someof the various embodiments, the transmit and receive (T_(x)-R_(x))system 402 may employ a multi-beam scan to further increase both imageprojection and object fine detail detection. In one or more of thevarious embodiments, the transmit and receive (T_(x)-R_(x)) system 402may employ a dual scanning system that can simultaneously scan twoparallel lines in the receive system 412 (“dual twin-matched epipolar”),which may result in double the resolution.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 402 may provide powerful three-dimensional motioncapture. In some of the various embodiments, the receive system 412 mayinstantaneously (or practically instantaneously) find a voxel range viatwo receivers of the receive system 412 (e.g., offset in an epipolarstereo configuration as explained above). For example, the receivesystem 412 may employ a hardware or software Look Up Table (LUT) todetermine the Z distance. In some of the various embodiments, separatethird sensor determines the vertical dimension (e.g., as explainedabove). In some of the various embodiments, one of the epipolarreceivers may provide a two-dimensional instantaneous read out. Forexample, the receive system 412 may determine the vertical dimension viatwo receivers (e.g., a one-dimensional receiver and a two-dimensionalreceiver).

In some of the various embodiments, the transmit system 404 may lack oneor more of a position sensor or a feedback loop to track instantaneousscan angles of the transmit system 404. The transmit system 404 mayinclude an ultra-fast open-loop MEMS laser scanning system (e.g., twofast resonant one-dimensional scanning mirrors configured in a relayoptics arrangement, such as, for example, the BTendo dual-mirror systemacquired by STmicroelectronics™ or Lissajous dual mirror scanningsystems developed by Fraunhofer™).

In one or more of the various embodiments, receive system 412 mayprovide ultra-wide fast scanning. For example, the receive system 412may include one or more “Lamb Drive” piezo-electric driven resonantsteel mirror systems that have been demonstrated recently in researchlaboratories in Tsukuba Japan. In some of the various embodiments, thereceive system 412 can scan wide angles at up to 60 kHz (120,000 linesper second). In some embodiments, the transmit and receive (T_(x)-R_(x))system 402 may provide a dual stereo or triple sensor arrangement toprovide a wide range of such novel scanning options. In one or more ofthe various embodiments, the receive system 412 may record and trackvarious points, flying spots, or scan lines with pin point accuracy inthree dimensions by an N (where N>=2) camera system. In some of thevarious embodiments, the receive system 412 may do so regardless ofquality, trajectory, or periodicity of optics of the transmit system412. For example, the transmit and receive (T_(x)-R_(x)) system 402 mayprovide pseudo-random three-dimensional surface scanning, for example,as described U.S. Pat. No., 8,430,512 assigned to PhotonJet Scanner.

In one or more of the various embodiments, the receive system 412 maydetermine and provide each voxel position as soon as one or more lightpulses are detected by a pair of pixels (e.g., one in a left receiverand one in a right receiver). In some of the various embodiments, thetransmit and receive (T_(x)-R_(x)) system 402 may optionally determineand provide each voxel position in response to the transmit and receive(T_(x)-R_(x)) system 402 pairing up the one or more detected lightpulses with a corresponding elevation value (e.g., one or more of aseparate receiver, a two-dimensional left receiver, a two-dimensionalright receiver, or the transmit system 404 can instantly compute a Zvalue).

Part of a light field of a reflected beam may reach each of threesensors simultaneously (or practically simultaneously). Three resultingsignals may be naturally synchronized. In one or more of the variousembodiments, the receive system 412 may unambiguously correlate (e.g.,matched up) these signals. When a stereo pair of pulses is detected, thereceive system 412 provides an instantaneous (or practicallyinstantaneous) triangulation, nearly instantaneously yieldingcorresponding X & Z values of a voxel. An additional elevation sensormay provide the elevation (e.g., Y value), for example, as describedabove. In some of the various embodiments, the receive system 412 maydetermine the elevation by an additional read out in one of the dualreceivers. In one or more of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 402 may maintain the sequential voxeldetection speed (e.g., up to one voxel per nanosecond) even at longerdistances (e.g., regardless of observation lag). In some of the variousembodiments, the receive system 412 may eliminate correlation of anoutgoing transmission angle with an incoming reception angle. In someembodiments, the receive system 412 may eliminate ambiguity caused by aninitially unknowable time of flight (e.g., unknown elapsed time betweenoutgoing and receiving). For farther objects, in response to thebaseline offset becoming too small of a fraction of a range, thetransmit and receive (T_(x)-R_(x)) system 402 may implement time offlight (e.g., by pulsing or modulating a laser source in a known codesequence, such as, for example, AM or FM Phase modulation, grey coding,or tapping beats). FIG. 18 illustrates an example procedure (e.g., alogical flow) of how the transmit and receive (T_(x)-R_(x)) system 402may determine when to range with time of flight (e.g., for far field)and when to use triangulation (e.g., for mid field or near field).

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 402 may provide high-speed three-dimensional motioncapture. In some of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 402 may produce accurate and fine-grained,high-speed three-dimensional motion maps of scanned objects. In someembodiments, the transmit and receive (T_(x)-R_(x)) system 402 mayoptionally provide one or more of the motion capture or the motion mapsin full color. The transmit and receive (T_(x)-R_(x)) system 402 mayinclude various robust high-speed color 3D motion capture systems (e.g.,as explained above) which produce accurate and fine-grained high-speed3D motion maps of scanned objects—optionally in full color.

In one or more of the various embodiments, the receive system 412 mayinclude one or more cameras for fast asynchronous triangulation. Lowcost so-called “rolling shutter” mobile cameras may have small (e.g., 1to 1.5 micron) pixels in 5 to 10 MP arrays. Each pixel (or single-photonavalanche diode (SPAD)) can serve a same column detection circuit. Rowsmay be selected, actively shuttered in a manner that may be analogous toCMOS camera pixels. In one or more of the various embodiments, however,fast and binary threshold detection of a flying spot may have a highpriority. In some of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 402 may determine a direction substantiallyorthogonal to a fast axis scan direction from feedback of a slow angleof one or more scanners (e.g., the vertical or elevation angle c). Insome embodiments, the transmit and receive (T_(x)-R_(x)) system 402 mayutilize these features to determine where a pixel has landed. Forexample, the transmit and receive (T_(x)-R_(x)) system 402 may utilizethese features to provide instantaneous (or practically instantaneous)(optionally asynchronous) column detection.

In one or more of the various embodiments, the receive system 412 mayinclude a 4-way receiver sensor system. In some of the variousembodiments, the 4-way receiver sensor system may include fourapertures. In some embodiments, the apertures may be arranged in asquare. In one or more of the various embodiments, the quad receiversensor system may have dual orthogonal disparity.

In one or more of the various embodiments, the receive system 412 maycapture and image the spot four times (e.g., each time with row andcolumn detection). In some of the various embodiments, at each spotrecording instance, the receive system 412 may record four azimuth Xvalues and four epsilon Y values (e.g., a total of 8 values, therebycapturing four times more light). In some embodiments, the receivesystem 412 may optionally apply one or more of R, G, B, or NIR filters.For example, each of the four quadrants may capture NIR (e.g., up toeight instantaneous (or practically instantaneous) readouts) for 4×signal strength).

In one or more of the various embodiments, the receive system 412 maydetermine depth estimation by scanned laser triangulation. In some ofthe various embodiments, the receive system 412 may determine the depthinformation based at least in part on the following formula:Z=hD/(Q−htanθ). The variables of this formula may represent thoseillustrated in the figures.

For example, as illustrated throughout the figures and as discussedabove, Z may represent a distance to a point where a beam was reflected(e.g., an apex C in a triangle ABC, where Z may be a distance that isorthogonal to the baseline (e.g., Z represents a height of the trianglemeasure as measured from C to base AB along a path that is orthogonal tobase AB). In some of the various embodiments, h may represent anorthogonal distance from a center of an aperture (e.g., a lens) to asensor plain. For example, as discussed above and as illustrated inFIGS. 4, 5, and 30, h may represent a focal length f of a lens system.In some of the various embodiments, the lens system may focus far awaylight to a small pixel-sized spot on the sensor surface 414. Asillustrated throughout the figures and as discussed above, D mayrepresent a baseline offset distance or baseline distance. For example,in an assisted stereo, the baseline is a distance between two receiversof a stereo pair (e.g., a left receiver and a right receiver). Asdiscussed above and as illustrated in FIGS. 4 and 5, Q may represent alateral (e.g., azimuthal) disparity as measured by, for example, adistance along the sensor plane from an optical center to where an imageof a scanning spot is projected onto the sensor 414. For example, Q maybe measured along a fast scanning direction (e.g., in some of thevarious embodiments, the receive system 412 may implement an idealepipolar arrangement in which directions of scanning rows in the sensor414 and the baseline are all parallel). As illustrated in FIG. 4, θ mayrepresent an angle from which an incoming ray deviates from a centraloptical axis of a receiver. For example, θ may be a complement of areceiver angle β (e.g., β+θ=90 degrees). In one or more of the variousembodiments, the transmit and receive (T_(x)-R_(x)) system 402 mayachieve greater precision in estimating Z by more finely measuring eachof these variables.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 402 may have a fixed baseline distance D. Forexample, the baseline distance D can be established with great precisionat manufacturing, through some kind of calibration. In some of thevarious embodiments, where a large baseline (e.g., 1.5 meters) isfeasible and desirable (e.g., in an automotive set up), there might besome recalibration to adjust for mechanical movements or misalignmentsthat occur naturally.

In one or more of the various embodiments, the focal length h may beprecisely measurable (e.g., in microns) after the lens and sensor 414have been assembled.

In one or more of the various embodiments, the incoming ray deviationangle Θ may be determined by measurement of a relative spot location inthe sensor 414. In some of the various embodiments, the receive system412 may approximate the incoming ray deviation angle Θ based on pixellocation nearest to a maximum illumination centroid of a spot (e.g., thelocation of optimal brightness) in the sensor. In some embodiments, thetransmit and receive (T_(x)-R_(x)) system 402 may providethree-dimensional measurements regardless of whether the spot is sharplyfocused. For example, the transmit and receive (T_(x)-R_(x)) system 402may provide three-dimensional measurements based on a clear center ofbrightness for the spot.

In one or more of the various embodiments, the receive system 412 mayhave precise (e.g., to the nanometer) and invariant pixel locations. Insome of the various embodiments, the receive system 412 may have pixellocations that may be fixed by lithography and fixedness of silicon. Insome embodiments, sizing and spacing of the pixels may define aquantization limit of this discrete location measurement.

In one or more of the various embodiments, the receive system 412 maybenefit from a smaller ratio of pixel size to sensor size. For example,the receive system 412 may benefit from an increase in pixel quantity.Smaller pixels, however, may receive less light. Small sensors (e.g., asmay result from the smaller pixels) may be much less expensive inpractice. These small sensors, however, may limit aperture size andfield of view. To gather enough light for sufficiently long rangetriangulation and ranging, larger apertures may be desirable, but theiroptics may provide more support for narrower fields of view (e.g.,governed by the law of etendue, by geometric optics, and practicalf-numbers of lenses).

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 402 may employ one or more methods to achieve acertain degree of “hyper resolution.” For example, the transmit andreceive (T_(x)-R_(x)) system 402 may very precisely determine an exact(or practically exact) arrival time of a sweeping beam, sweeping with aknown velocity on a surface (e.g., contiguous surface trajectoryvelocity).

Large apertures may result in large sensors with larger pixels that maybe expensive. For example, a modern high-resolution camera such as the42 Megapixel SLR from Sony™ may accommodate ultra low-light capture withlarge apertures but may employ 3×3 Micron pixels that may involvethousands of dollars of optics and sensors. CMOS sensor technology hasmade great progress as evidenced by extremely powerful optics beingavailable at consumer prices.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 402 may employ latest-generation CMOS sensors innovel ways to create machine vision systems of utmost speed, resolution,and sensitivity. For example, the transmit and receive (T_(x)-R_(x))system 402 may provide three-dimensional metrology and ultra-fastthree-dimensional scanning systems (e.g., LIDAR Triangulation and HybridLIDAR-Triangulation).

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 402 may leverage low-cost mobile camera pixeltechnology with 1.5 or 1 micron pixels to provide a camera sensor thatmay be 10 to 100 times smaller and cheaper. In some of the variousembodiments, the transmit and receive (T_(x)-R_(x)) system 402 mayposition fine pitch pixels in strategic locations. In some embodiments,the transmit and receive (T_(x)-R_(x)) system 402 may control a field ofview of individual sensors to provide big apertures. In one or more ofthe various embodiments, the transmit and receive (T_(x)-R_(x)) system402 may use a plurality of cheap sensors and mass producible opticswhile improving field of view, focal length, and depth of field. Forexample, rather than using one large collection type optics that mayinclude a sensor costing well over $1,000 (e.g., DSLR Alpha a7R II fromSony™ with 42.4 M Pixel sensor), the transmit and receive (T_(x)-R_(x))system 402 may use an array of low cost mobile phone sensors (e.g., $5for a diffraction limited 10M pixel camera) or mobile phone sensors. Insome of the various embodiments, the transmit and receive (T_(x)-R_(x))system 402 may include an array of such sensors, wherein the sensorshave slightly modified specifications as explained herein. In one ormore of the various embodiments, the transmit and receive (T_(x)-R_(x))system 402 may detect 100 Mega pixel composite images for less than$100.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 402 may include in stereo or N camera multi-viewarrangements. In some of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 402 may utilize overlapping regions forextra three-dimensional (e.g., Z-range) resolution. In some embodiments,the transmit and receive (T_(x)-R_(x)) system 402 may fiducially anchorboth observation (e.g., each three-dimensional voxel on an object'ssurface) and observer (e.g., camera). For example, the transmit andreceive (T_(x)-R_(x)) system 402 may utilize each pixel in the field ofview, when illuminated by pinprick flash scanning lasers, as “absoluteground truth.” In one or more of the various embodiments, the transmitand receive (T_(x)-R_(x)) system 402 may anchor one or more of theobservation or the observer in three-dimensional space and anchoring theone or more of the observation or the observer with nanosecond-precisionin time.

Illustrated Circuitry of the Sensing Systems

In one or more of the various embodiments, each pixel of the sensor 414of the receive system 412 may include one or more photo diodes (PD). Forexample, each pixel may include one or more pinned photo diodes (PDD).In some of the various embodiments, the receive system 412 may includeone or more transfer gate transistors (TG) (e.g., the receive system 412may include a transfer gate transistor for each photo diode). Forexample, a photo diode in a pixel may selectively be enabled in responseto a transfer signal that a transfer gate transistor associated with thephoto diode receives from a transfer gate enable line. In someembodiments, a transfer gate transistor may provide a high (e.g.,enabled) transfer signal to a row of pixels. Photodiodes of the row ofpixels may be configured and arranged to, in response to the high (e.g.,enabled) transfer signal, become set to “spew” photo-electrons (e.g.,set to source an instantaneous spike of photocurrent) that trigger anop-amp (e.g., a specially configured transistor circuit) to “jolt” acolumn sense line that belongs to the particular pixels of thephotodiodes.

For example, one or more of the op-amp logic or the pixel-leveltransistor logic may have a threshold function. In some of the variousembodiments, the logic may prevent the jolt from occurring unless thephoto-current exceeds an instantaneous peak value. For example, a weakpull-up provided by a weakly pulled up reset transistor RST may keep theoutput of the transfer gate transistor (e.g., photodiode output) high inthe absence of photo current until a certain peak instantaneousphoto-electron supply is strong enough (and a pull up resistor value islarge/weak enough) to lower a voltage provided to the op-amp for a timethat is long enough to raise the jolt to be triggered (e.g., the op-ampswitches momentarily to connect V_(DD) to a column trigger detector,such as, for example, a column sensing line). An example of such acircuit function is explained in further detail below.

In one or more of the various embodiments, the sensor 414 may employ aninstantaneous asynchronous two-dimensional row and column function. Insome of the various embodiments, the “jolt” signal may be provided toboth row and column sense lines corresponding to the pixel. For example,the receive system 412 may include fast detection logic at each of thecolumns and rows (e.g., the X value (such as, for example, azimuth,beta) and Y value (such as, for example, epsilon)).

In some embodiments, the receive system 412 may determine each voxel ateach nanosecond with a minimal latency at a rate up to 1 billion XYpairs per second.

In one or more of the various embodiments, the receive system 412 mayinclude one or more dual-function sensors that provide both asynchronouspeak flow detection and row time exposure integration camera function.In some of the various embodiments, the dual-function sensor may provideinstantaneous position via lighting detection and/or logarithmic bypasswith a longer integration function provided by a four-transistor pinnedphoto-diode pixel. For example, the op-amp may temporarily deflect(e.g., re-route) a rush of photo-electrons caused by an intense andinstantaneous (or practically instantaneous) (e.g., temporally andspatially sharp) spot transition, and the receive system 412 maypreserve the four-transistor pinned photo-diode to continue flowingsub-threshold photo current into a floating gate to maintain synchronousrolling or global shutter camera function.

In one or more of the various embodiments, the receive system 412 mayoptionally include NIR peak detection circuits and sense lines burieddeep below “shallower” RGB camera circuits (e.g., using lower subsurface metal lines, or, in case of back illumination, coarser surfacestructure).

In one or more of the various embodiments, the receive system 412 mayprovide pixel specific output proportional to a natural logarithm of aninstantaneous photo-current produced by a pixel in response to detectinga laser light reflected from a three-dimensional object. For example, aphotodiode of the receive system 412 may generate photo-current that maybe represented by the following relationship: I_(ph)≈e^(Vph/V)T where“I_(ph)” represents photo current of the photodiode, “V_(ph)” representsvoltage across the photodiode, and “V_(T)” represents thermal voltage.In some of the various embodiments, a pixel circuit of the receivesystem 412 may provide an output that drives a column sense line. Forexample, the output of the pixel circuit may be proportional to anatural logarithm of an instantaneous photodiode current I_(ph).

In one or more of the various embodiments, the receive system 412 mayimplement the 4-way receiver sensor system in multiple roles. Forexample, the receive system 412 may implement the 4-way receiver sensorsystem in a traditional four-transistor PPD pixel role, integratingphoto current over a longer exposure period. In some of the variousembodiments, the receive system 412 may implement the 4-way receiversensor system in an alternative mode of operation to immediately relay aphoton pulse with minimal delay (e.g., nanoseconds) towards an adjacentcolumn sense line for a given pixel.

FIG. 24 illustrates an exemplary sensor portion 2400 that includes anexemplary four-transistor pinned photodiode (PDD) pixel 2402. The sensorportion 2400 may be that of a sensor of a receive system. For example,the receiver may be the same as or similar to one or more of thoseexplained above. In one or more of the various embodiments, the pixel2402 may employ logic 2404 that provides one or more of row-select orrow-control. In some of the various embodiments, the pixel 2402 mayinclude a column sense line 2406 over which the pixel 2402 outputs data(e.g., to a column decoder or analog to digital converter).

In one or more of the various embodiments, the pixel 2402 may include atransfer gate 2408. In some embodiments, the transfer gate 2408 mayconnect a pinned photodiode 2410 of the pixel 2402 to a floatingdiffusion well 2412 of the pixel 2402. In some of the variousembodiments, the PPD 2410 may provide a charge to the transfer gate 2408in response to capturing photons. In some of the embodiments, thetransfer gate 2408 may provide the charge to the floating diffusion well2412. In one or more of the various embodiments, the floating diffusionwell 2412 may hold the charge. In some of the various embodiments, avoltage may fall as photoelectrons arrive through the transfer gate2408. For example, the voltage may fall inversely proportionally withrespect to previously received photon flux.

In one or more of the various embodiments, a pixel row that contains orbears the pixel 2402 may include a reset circuit 2414. In some of thevarious embodiments, the reset circuit 2414 may reset the floatingdiffusion well 2412 to V_(DD) in response to one or more pulses that thereset circuit 2414 receives over a reset line 2416. For example, inresponse to the one or more pulses, the reset circuit 2414 may reset awhole row of pixels before exposure. In one or more of the variousembodiments, the transfer gate 2408 may receive one or more signals overa transfer gate enable line 2418. In some of the various embodiments,pixel integration may be enabled in response to the transfer gate 2408receiving the one or more signals over the transfer gate enable line2418.

In one or more of the various embodiments, the pixel 2402 may include aread enable circuit 2420. In some of the various embodiments, the readenable circuit 2420 may enable a pixel read (e.g., a reduced voltageafter exposure) over the column sense line 2406 for the pixel 2402. Inone or more of the various embodiments, the read enable circuit 2420 mayreceive one or more signals over a read enable line 2422. For example,the read enable circuit 2420 may enable the pixel read in response toreceiving the one or more signals over the read enable line 2422 (e.g.,connecting to the column decoder or analog to digital converter). Insome embodiments, the pixel row that contains the pixel 2402 may includethe read enable circuit 2420. In some of the various embodiments, theread enable circuit 2420 may provide the pixel read enable for eachpixel of the pixel row in response to receiving the one or more signalsover the read enable line 2422. For example, the read enable circuit2420 may enable the pixel read for each pixel in the row by respectivecolumn sense lines that correspond to each pixel (e.g., connecting torespective column decoders or analog to digital converters).

FIG. 25 shows an exemplary sensor portion 2500 that includes anexemplary two-transistor photodiode pixel 2502. The sensor portion 2500may be that of a sensor of a receive system. For example, the receivermay be the same as or similar to one or more of those explained above.In one or more of the various embodiments, the pixel 2502 may employlogic 2504 that provides one or more of row-select or row-control. Insome of the various embodiments, the pixel 2502 may include a columnsense line 2506 over which the pixel 2502 outputs data (e.g., to acolumn decoder or analog to digital converter).

In one or more of the various embodiments, the pixel 2502 may include atransfer gate 2508. In some of the various embodiments, the transfergate 2508 may connect a pinned photodiode 2510 of the pixel 2502 to afloating diffusion gate 2512 of the pixel 2502. In some of the variousembodiments, in contrast to the reset circuit 2414 of FIG. 24, the pixel2502 may include a weak pull-up resistor (R). In some embodiments, theweak pull-up resistor may keep the floating diffusion gate 2512 pulledup to V_(DD) when no photo-electron pulse is detected. Alternatively, athird reset transistor may be set to act as a weak pullup that can becontrolled by an external reset control line. In one or more of thevarious embodiments, the transfer gate 2508 may receive one or moresignals over a transfer gate enable line 2514. In some of the variousembodiments, pixel or row activation (e.g., light sensing) may beenabled in response to the transfer gate 2508 receiving the one or moresignals over the transfer gate enable line 2514.

For example, at time to photons may rush into the pixel photodiode 2510.The pulse of photodiode-generated electrons may rush through thetransfer gate 2508 (e.g., at ti). At t₂, the column sense line 2506 maybe pulled low (e.g., a peak electron flux current may flow from thephotodiode 2510 and through the transfer gate 2508 and may be strongerthan a weak pull up provided by the pull-up resistor) to produce asignal. At t₃ a column sensing circuit 2516 may amplify the signal. Insome of the various embodiments, each column may have a correspondingcolumn sensing circuit (e.g., for column sense lines 2506 through columnsense line N that connects to an Nth column sensing circuit 2518). Basedon the signal, the receive system may instantly (or practicallyinstantly) know in which column the photon pulse occurred. In one ormore of the various embodiments, the receive system may have low signalpropagation lag (e.g., in the order of nanoseconds). In some of thevarious embodiments, the receive system may have small pixel structures,capacitances, and impedances. In some embodiments, the column sense line2506 may have its own amplifier (e.g., column sensing circuit 2516).

In one or more of the various embodiments, the receive system may encodethe signal with column numbers. In some of the various embodiments, thereceive system may combine the signals for various columns into a serialbus. In some embodiments, the receive system may implement sense linelogic to encode the signals with column numbers. Additionally oralternatively, the receive system may implement the sense line logic tocombine the signals for various columns into the serial bus.

In one or more of the various embodiments, the receive system mayprovide a signal for each detected scan beam reflection. For example,the receive system may implement a 1 GHz clock to provide asynchronousvoxel registrations (e.g., up to 1 billion voxels per second).

FIG. 26 shows an exemplary flashed illumination 2600 that employsexemplary wave-front color separation. An exemplary transmit and receive(T_(x)-R_(x)) system may implement the flashed illumination 2600. Forexample, the transmit and receive (T_(x)-R_(x)) system may be the sameas or similar to one or more of those explained above. In one or more ofthe various embodiments, a transmit system of the transmit and receive(T_(x)-R_(x)) system may emit the flashed illumination 2600 in a scandirection 2602. In some of the various embodiments, the flashedillumination 2600 may include a “PlainBow” Flashed Phosphor illuminationwave front color separation with short (e.g., UV or blue) wavelengths2604 leading and long (e.g., red or NIR) wavelengths trailing 2606.

FIG. 27 illustrates an exemplary cascaded trigger pixel system 2700 thatemploys exemplary separate sense lines to sequentially capture variousexemplary color-separated and time-separated components. An exemplarysensor of an exemplary receive system may include the cascaded triggerpixel system 2700. For example, the receive system may be the same as orsimilar to one or more of those explained above. In one or more of thevarious embodiments, the separate sense lines of the cascaded triggerpixel system may sequentially capture one or more of the color-separatedand time-separated components of the flashed illumination 2600 of FIG.26.

In one or more of the various embodiments, the cascaded trigger pixelsystem 2700 may include a first transfer gate 2702 that connects to afirst photodiode 2704, a second transfer gate 2706 that connects to asecond photodiode 2708, a third transfer gate 2710 that connects to athird photodiode 2712, and a fourth transfer gate 2714 that connects toa fourth photodiode 2716. In some of the various embodiments, the firsttransfer gate 2702 may receive one or more signals over a transfer gateenable line 2718 at to. In some embodiments, the first transfer gate2702 may pass a first signal from the first photodiode 2704 over a firsttransfer gate output line 2720 in response to receiving the one or moresignals over the transfer gate enable line 2718 and the first photodiode2704 capturing a light spot. The first transfer gate output line 2720may deliver the first signal to the second transfer gate 2706 and to oneor more row or column sense lines at t₁ (e.g., t₁ may equal t₀+Δ₁). Thesecond transfer gate 2706 may pass a second signal from the secondphotodiode 2708 over a second transfer gate output line 2722 in responseto receiving the first signal and the second photodiode 2708 capturing alight spot. The second transfer gate output line 2722 may deliver thesecond signal to the third transfer gate 2710 and to one or more row orcolumn sense lines at t₂ (e.g., t₂ may equal t₁+Δ₂). The third transfergate 2710 may pass a third signal from the third photodiode 2712 over athird transfer gate output line 2724 in response to receiving the secondsignal and the fourth photodiode 2712 capturing a light spot. The thirdtransfer gate output line 2724 may deliver the third signal to thefourth transfer gate 2714 and to one or more row or column sense linesat t₃ (e.g., t₃ may equal t₂+Δ₃). The fourth transfer gate 2714 may passa fourth signal from the fourth photodiode 2716 over a fourth transfergate output line 2726 in response to receiving the third signal and thefourth photodiode 2712 capturing a light spot. The fourth transfer gateoutput line 2726 may deliver the fourth signal to a subsequent transfergate (not shown) and to one or more row or column sense lines at t₄(e.g., t₄ may equal t₃+Δ₄).

FIG. 28 shows an exemplary flash-triggered four-transistor photodiodepixel 2800 (e.g., a pinned photodiode pixel). An exemplary sensor of anexemplary receive system may include the flash-triggered four-transistorphotodiode pixel 2800. For example, the receive system may be the sameas or similar to one or more of those explained above. In one or more ofthe various embodiments, the flash-triggered four-transistor photodiodepixel 2800 may employ logic 2802 that provides one or more of row-selector row-control. In some of the various embodiments, the pixel 2800 mayinclude a column sense line 2804 over which the pixel 2800 outputs data(e.g., to a column decoder or analog to digital converter).

In one or more of the various embodiments, the pixel 2800 may include atransfer gate 2806 that passes current from a photodiode 2808 to afloating diffusion well 2810 of the pixel 2800. In some of the variousembodiments, a pixel row that contains or bears the pixel 2800 may havea reset circuit 2812 that receives one or more signals over a reset line2814. In some embodiments, the transfer gate 2806 may enable in responseto receiving one or more signals over a transfer gate enable line 2816.In some of the various embodiments, the pixel 2800 may include a resetenable circuit 2818 that enables a pixel read over the column sense line2804 in response to receiving one or more signals over a read enableline 2820.

In one or more of the various embodiments, the pixel 2800 employ one ormore lines of the logic 2802 to sequentially capture variouscolor-separated and time-separated components. In some of the variousembodiments, the same one or more lines may provide fast-sequentialpixel-read outs with regard to one or more captures of one or more ofthe color-separated and time-separated components of the flashedillumination of FIG. 26.

Additional Illustrated Aspects of the Sensing Systems

As shown in FIG. 29, an exemplary transmit and receive (T_(x)-R_(x))system 2900 may employ exemplary LIDAR triangulation (e.g., as explainedabove). For example, the transmit and receive (T_(x)-R_(x)) system 2900may be the same as or similar to one or more of those explained above.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 2900 may include a stereo pair of triangulatingLIDAR receivers 2902 and 2904. In some of the various embodiments, thetransmit and receive (T_(x)-R_(x)) system 2900 may include a scanninglaser transmitter 2906.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 2900 may offset the two receivers 2902 and 2904from the scanning laser transmitter 2906 (e.g., along a baseline of thetransmit and receive (T_(x)-R_(x)) system 2900). In some of the variousembodiments, the scanning laser transmitter 2906 may be arranged to scanacross a field of view of the transmit and receive (T_(x)-R_(x)) system2900. In some embodiments, the scanning laser transmitter 2906 may bearranged to sweep a scanning beam 2908 in successive scans. In one ormore of the various embodiments, the transmit and receive (T_(x)-R_(x))system 2900 may arrange the receivers 2902 and 2904 to have respectivecentral optical axes 2910 and 2912 that are parallel with each other. Insome of the various embodiments, the central optical axes 2910 and 2912may be perpendicular to the baseline of the transmit and receive(T_(x)-R_(x)) system 2900.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 2900 may be a smart hybrid triangulating LIDARsystem. In some of the various embodiments, the smart hybridtriangulating LIDAR system may optimize ambient illuminationsuppression. For example, the smart hybrid triangulating LIDAR systemmay employ synchronized time-selective triggered pixel activation. Inone or more of the various embodiments, the scanner may project imagesof scan lines onto surfaces in the field of view. In some of the variousembodiments, the receivers 2902 and 2904 and the scanner 2906 may bearranged to align the images with rows in sensors of the receivers 2902and 2904. For example, the receivers 2902 and 2904 and the scanner 2906may be arranged in an epipolar configuration.

In one or more of the various embodiments, the scanning lasertransmitter 2906 may include a light source and a scanning mirror. Insome of the various embodiments, the light source may emit shortcollimated photonic pulses as the scanning mirror scans across the fieldof view. In some embodiments, the receivers 2902 and 2904 and thescanner 2906 may be arranged so that trajectories of reflections ofthese pulses trace along successive rows in each of the sensors. In oneor more of the various embodiments, the collimated photonic pulsesemitted by the scanner may include, for example, “tracer bullets” (e.g.,as described in U.S. Pat. Nos. 8,282,222, 8,430,512, 8,696,141, and8,711,370 and U.S. patent application Ser. No. 14/823,668, each of whichis assigned to PhotonJet). In one or more of the various embodiments,the scanning laser transmitter may project time-of-flight tracerbullets. In some of the various embodiments, the stereo pair oftriangulating time-of-flight receivers may capture reflections of thetime-of-flight tracer bullets. In some embodiments, projected light mayreflect on a surface in the field of view. A fraction of the reflectionmay return towards the system. The receivers may capture the fraction ofthe reflection.

In one or more of the various embodiments, the smart hybridtriangulating LIDAR system may combine triangulation with time-of-flightranging. In some of the various embodiments, the transmitter 2906 maytransmit the tracer bullets along a known trajectory axis (e.g., raydirection with angular coordinates α & ε) at speed c (approx. 3×10⁸m/sec). In some embodiments, the transmitter 2906 may emit the tracerbullets along the straight line by reflecting the tracer bullets off thescan mirror.

In one or more of the various embodiments, in response to the tracerbullets impacting a surface at distance Z (e.g., a first distance 2914,a second distance 2918, or a third distance 2916), a fraction of photonsof the tracer bullets may reflect back toward the three-dimensionalsensing system 900 as explained above. In some of the variousembodiments, apertures of the receivers 2902 and 2904 may capture thefraction of the photons as explained above. In some embodiments, thereceivers 2902 and 2904 may be offset at distance D from each other asexplained above. In one or more of the various embodiments, the scanningsource transmitter 2906 may be positioned in a center of the baseline.In some of the various embodiments, each receiver may be positioned at adistance D/2 from the scanning source transmitter 2906.

Where the receivers 2902 and 2904 are offset from the transmitter 2906,the photons may travel in triangles to reach the receivers 2902 and2904. These triangles may become more acute as distance Z increases. Inone or more of the various embodiments, as Z increases, returninglight's chief ray may rotate out toward a vanishing point. In some ofthe various embodiments, as the returning light's chief ray rotates outtoward the vanishing point, an angle beta β may increase. In someembodiments, the increase in the angle beta β may converge the outgoingbeam 2908 and incoming reflected light's chief ray directions towardbeing parallel (e.g., as the Z increases toward infinity, α≈β). In oneor more of the various embodiments, one or more columns in the sensormay capture each incoming ray (e.g., each ray with incoming angle β).For example, each incoming ray may be captured by one or more columns atan intersection point that may slide along a row toward a center of thesensor (e.g., along the epipolar axis).

In one or more of the various embodiments, the sensor may identify aunique pixel location for each distance Z that the outgoing ray travelsalong the outgoing ray direction α. In some of the various embodiments,the incoming ray angle β may be a function of Z and α. In someembodiments, the transmit and receive (T_(x)-R_(x)) system 2900 mayemploy a time of flight for the reflected photons. For example, the timeat which the reflected photons arrive at the sensor location is given byt_(ToF) (e.g., a round trip time at a speed of light wheret_(ToF)=2*Z/c).

In one or more of the various embodiments, the smart three-dimensionaltriangulating LIDAR system may know where to expect the photons toreturn (e.g., an anticipated location in the sensor). In some of thevarious embodiments, the smart three-dimensional triangulating LIDARsystem may know when to expect the photons to return. In someembodiments, the anticipated location in the sensor may be a function oftime. For example, the anticipated location may be a function of thetracer bullet's time of flight (which may be a function of Z). In one ormore of the various embodiments, based on this fixed physical geometricrelationship, the smart LIDAR system may preemptively correlate aninstantaneous moment and location of the returning photons.

In one or more of the various embodiments, the system 2900 may knowthat, for each possible distance Z, there might be a reflecting surfaceor obstacle in a path of the beam. In one or more of the variousembodiments, the system 2900 may know, for each such possible Z value,exactly (or practically exactly) when and where in the sensor thephotons should return. In actuality, each tracer bullet may reflect forone value of Z and return in one location in the sensor at a given time.Because the system 2900 may anticipate each possibility and because apossible photon landing at a location in the sensor can happen at aunique moment (e.g., one possible location at a given time), the system2900 can open up a “landing” pixel as a function of time. In one or moreof the various embodiments, the system 2900 may slide the position ofthe “landing pixel” very quickly (e.g., in nanoseconds) along a row inthe sensor.

In one or more of the various embodiments, the system 2900 may, at agiven time, activate a very small subsection of a row (e.g., one pixelor a few pixels). In some of the various embodiments, the system 2900may slide a subsection window of activated pixels along the row (e.g.,within a microsecond from a point of greatest disparity d_(max) to apoint of least disparity d_(min)).

In one or more of the various embodiments, the transmitter 2906 mayinclude a pixel sequential transmitter. In some of the variousembodiments, the pixel sequential transmitter may launch a brief (100ps), narrowly collimated (<500 microns) “spit ball” of photons toward atarget (e.g., Z feet away). The photons may splash onto the target(e.g., Z nanoseconds after launch). Some of the photons may reflect backin a direction of one of the receivers 2902 and 2904 as explained above.After another Z ns, the reflected photons may enter an aperture of thereceivers 2902 and 2904. The receivers 2902 and 2904 may collimate thereflected photons to a point that may approximately match a size of thepixels (e.g., a micron) on the sensor's surface.

In one or more of the various embodiments, when the receivers 2902 and2904 simultaneously detect reflected photons of a photon bullet, thetransmit and receive (T_(x)-R_(x)) system 2900 may associate thosedetections with each other, determining a correspondence between one ormore pixels in the receiver 2902 and one or more pixels in the receiver2904 in this stereo detection (e.g., as shown by arrow 2920). In thismanner, the transmit and receive (T_(x)-R_(x)) system 2900 mayinstantaneously and unambiguously provide the correspondence between thepixels when using a pixel sequential (e.g., twitch pixels, SPAD, or thelike) camera.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 2900 may have expected the reflected photons toarrive at the point at that particular time. In some of the variousembodiments, the transmit and receive (T_(x)-R_(x)) system 2900 may openan ultra-fast shutter that may open up a particular pixel at the point(e.g., just before the reflected photons arrive at the particularpixel). For example, the transmit and receive (T_(x)-R_(x)) system 2900may open up a sliding activated pixel window at a sliding position alongthe row.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 2900 may instantly (or practically instantly) setdetection circuits to an exact (or practically exact) sensitivity topositively detect an amount of the reflected photons that arrive at theparticular pixel. In some of the various embodiments, the receivers 2902and 2904 may include fast-gated logic that may activate column-sensinglogic. In some embodiments, this column-sensing logic activates asensing decoder for brief time period (e.g., at a moment of arrival of aspike of photocurrent generated in the particular pixel that thereflected tracer bullet activates). In some embodiments, the transmitand receive (T_(x)-R_(x)) system 2900 may synchronously match up theincoming electrons on arrival. In some embodiments, the receivers 2902and 2904 may amplify a corresponding signal to a strength sufficient toreach a control system.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 2900 may account for which pixel should capture thereflected photons at what time as explained above. In some of thevarious embodiments, the transmit and receive (T_(x)-R_(x)) system 2900may make small adjustments (e.g., to a fraction of a nanosecond) to, forexample, account for known system latency (e.g., lag between arrival ofphotons and a boosted signal arriving at the control system). In someembodiments, the transmit and receive (T_(x)-R_(x)) system 2900 may,with an expected time-of-flight delay and disparity d confirmed,instantly (or practically instantly) compute (e.g., by looking up aprecomputed triangulation matrix) an exact (or practically exact)distance Z, for location X, Y (column and row address of the pixel thatthe reflected photons activate).

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 2900 may employ precise spatio-temporal singlepixel confinement. In some of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 2900 may activate a pixel just beforearrival of the reflected tracer bullet's photons at the pixel (e.g.,after a flight time that equates to 2*Z/c). In some embodiments, thetransmit and receive (T_(x)-R_(x)) system 2900 may do so by sequentiallyactivating pixels along a particular row that an active column decodersupports. In one or more of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 2900 may shutter out other light. In someof the various embodiments, the transmit and receive (T_(x)-R_(x))system 2900 may sequentially and very briefly activate the pixels alongthat row in an anticipatory fashion as explained above.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 2900 may preselect (e.g., both spatially andtemporally) potentially receiving pixels in the sensor. For example, thetransmit and receive (T_(x)-R_(x)) system 2900 may employ an epipolararrangement with a sliding pixel shutter as explained above. In some ofthe various embodiments, the transmit and receive (T_(x)-R_(x)) system2900 may detect the returning photons with pinpoint precision. In someembodiments, the transmit and receive (T_(x)-R_(x)) system 2900 may, inan anticipatory fashion, confine the moment and location of arrival ofthe reflected photons to as little as a single nanosecond and a singlepixel.

The following example illustrates the above. The following example usesround numbers for convenience, simplicity, and clarity. As shown byarrow 2920, simultaneously activated pixels in the receivers 2902 and2904 may have a disparity relative to each other (e.g., d_(min),d_((t)), and d_(max)).

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 2900 may use a laser beam to scan an areaequivalent to a single pixel in the system's field of view (e.g., a 10degree field of view) in 1 nanosecond. The sensor may include 10,000pixels in each row. Each pixel may capture 1/1000 of a degree of thefield of view. The transmit and receive (T_(x)-R_(x)) system 2900 mayscan each line of the field of view in 10 microseconds.

For example, the transmit and receive (T_(x)-R_(x)) system 2900 may“fire” an intense 100 pico second tracer bullet in a direction ofepsilon ε and alpha α. Shortly afterwards, the transmit and receive(T_(x)-R_(x)) system 2900 may activate one pixel in each receiver 2902and 2904 for approximately 1 nanosecond (e.g., one clock). The transmitand receive (T_(x)-R_(x)) system 2900 may select the pixel based on aposition of the pixel. For example, the transmit and receive(T_(x)-R_(x)) system 2900 may select the pixel because this pixelposition may be a position of maximum disparity d_(max) (e.g., greateststereo differential angle β-α) and may correspond to a minimal rangeZ_(min). The transmit and receive (T_(x)-R_(x)) system 2900 may shutterthe pixel off again and may activate a pixel neighbor of that pixel(e.g., may activate a pixel in the next column over) for approximatelyone nanosecond. The transmit and receive (T_(x)-R_(x)) system 2900 maysequentially proceed to each subsequent pixel until a maximum detectionrange position has been reached at d_(min).

As explained above, the maximum disparity, d_(max), may occur at theminimum range Z_(min), and the minimum disparity, d_(min), may occur atthe maximum range, Z_(max). The farther the light travels, the less thedisparity as explained above. At some point, the maximum observablerange, Z_(max), is reached. In response to reaching the maximumobservable range, Z_(max), the transmit and receive (T_(x)-R_(x)) system2900 may stop looking at the next pixel in the row. For example, ifthere are no objects (e.g., no reflecting surfaces) within this range orif the surface reflects an insufficient portion of the tracer bullet'slight, then the transmit and receive (T_(x)-R_(x)) system 2900 may failto record an event. In response to the transmit and receive(T_(x)-R_(x)) system 2900 failing to record an event, the system may“time-out.” In response to the transmit and receive (T_(x)-R_(x)) system2900 timing out (or approaching the time-out), the transmit and receive(T_(x)-R_(x)) system 2900 may fire a subsequent tracer bullet (e.g.,after maximum range time of 1 microsecond, a 500 ft. range, forexample).

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 2900 may employ a fast dynamical scanning mode(e.g., a mode in which the transmit and receive (T_(x)-R_(x)) system2900 may provide a fast dynamical voxel detection rate). In the dynamicscanning mode, the system may send a subsequent bullet in response torecording a given bullet's reflection. For short-range detection, aninverse-square law may indicate that the transmit and receive(T_(x)-R_(x)) system 2900 may need quadratically less energy in eachphoton bullet, while still achieving a sufficiency of photons arrivingin the sensor, as compared to energy in a photon bullet that travels toand from a target that spaced one further unit of distance from thetransmit and receive (T_(x)-R_(x)) system 2900 (e.g., near objectsautomatically receive a dense hail of low energy tracer bullets). Theshort distance allows a rapid firing of low energy tracer bulletswithout causing ambiguity in the sensor.

At larger distances (e.g., when detection is limited to a relativelynarrow range, such as, for example, a 20′ sensing range at 200′distance), the system may time out sooner and may, for example, fire asubsequent tracer bullet upon reaching a time period between bulletfirings that may corresponds to a time interval where simultaneouslyarriving reflections may cause ambiguity (e.g., 40 nanoseconds). Forexample, at a detection range setting of 20 feet and at a distance of200′, the transmit and receive (T_(x)-R_(x)) system 2900 may reach ratesup to 25M Voxels/sec.

The transmit and receive (T_(x)-R_(x)) system 2900 may dynamicallyamplify signals. Because closer objects may reflect more strongly thanfurther objects (e.g., according to the inverse-square law), an incomingsignal of reflected photons may be stronger and much easier to detect.In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 2900 may, therefore, provide less signalamplification for pixels associated with close range reflections (e.g.,both in those pixels and in circuits that connect to those pixels). Insome of the various embodiments, the transmit and receive (T_(x)-R_(x))system 2900 may include amplification circuits built in to column (orrow) sensing lines. In some embodiments, the transmit and receive(T_(x)-R_(x)) system 2900 may set the amplification of column sensinglines connected to pixels that are set to receive nearby reflections(e.g., pixels with current disparity values d close to dmax) totemporarily amplify by reduced magnitudes. In some embodiments, thetransmit and receive (T_(x)-R_(x)) system 2900 may set the amplificationof column sensing lines associated with pixels that receive longer rangesignals (e.g., with Z closer to Z_(max)), such as, for example, pixelsin positions where the disparity currently is closer to d_(min), totemporarily amplify by increased magnitudes. In some embodiments, thetransmit and receive (T_(x)-R_(x)) system 2900 may adjust, in real time,pixel circuits themselves (e.g., by adjusting photodiode bias or anadjustable gain circuit in pixel logic as illustrated in FIG. 38).

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 2900 may, in response to detecting a surface andestablishing an approximate range, adjust all three settings: the photonbullet energy, the repeat frequency, as well as the sensor sensitivity.In some of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 2900 may automatically set an optimal setting foreach situation.

TABLE 1 Range (R) 100′ 200′ 300′ d (1/R) 300 150 100 Photon (1/R²) 900300 100 t_(ToF) (in ns) 200 400 600

TABLE 2 Range 10′ 20′ 40′ 100′ ≅1/R² 1.00 .250 .0625 .010 Photons 1000250 62 10 Gain 1 4 16 100 Net signal 1.0 1.0 1.0 1.0 d (≈1/R) 100 50 2510 t_(ToF) (in ns) 20 40 80 200

In one or more of the various embodiments, a “twitchy pixel” mayinstantly (or practically instantly) report an incoming pulse ofreflected tracer bullet photons (e.g., via one or more of a highsensitivity sensing amplifier or a source follower connected to aphotodiode in the pixel). In some of the various embodiments, the pixelmay be arranged to cause an amplified pulse to travel via a column senseline to a gated sense amplifier. In some embodiments, this amplifier mayserve one column. The transmit and receive (T_(x)-R_(x)) system 2900 mayturn this amplifier on for a very brief moment (e.g., at timet=t_(ToF)=2*Z/c), for example, an exact (or practically exact) moment(one nanosecond) when the photon bullet's reflection arrives. In someembodiments, the system may determine Z within an accuracy of ½ of onefoot.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 2900 may provide sequential voxel scanning ofclose-range surfaces (e.g., surfaces within 10 feet) at, for example, 20nanoseconds per sequential location. In some of the various embodiments,the transmit and receive (T_(x)-R_(x)) system 2900 may scan up to, forexample, 50 Million Voxels per second (50 MVps).

FIG. 30 illustrates an exemplary LIDAR triangulating transmit andreceive (T_(x)-R_(x)) system 3000. For example, the transmit and receive(T_(x)-R_(x)) system 3000 may be the same as or similar to one or moreof those explained above. In one or more of the various embodiments, thetransmit and receive (T_(x)-R_(x)) system 3000 may include atriangulating transmitter-receiver pair 3002 and 3004. In some of thevarious embodiments, the transmit and receive (T_(x)-R_(x)) system 3000may include a photon transmitter 3002 at base point A of a triangle ABC.In some embodiments, the transmit and receive (T_(x)-R_(x)) system 3000may include a receiver 3004 at another base point B at an offset basedistance D (e.g., AB=D). The transmitter 3002 may emit a scanning beam3012 that reflects off a surface 3006 as reflected light 3014 toward thereceiver 3004 (e.g., a pixel 3016 of a sensor of the receiver 3004 maycapture the reflected light 3014, the pixel 3016 being at point i wherethe column position or number is proportional to the angle of beta β).

In one or more of the various embodiments, the receiver 3004 may includea sensor 3018. In some of the various embodiments, the sensor 3018 maybe arranged in a plane parallel to base AB at a distance f, where f isan effective focal length of a lens or focusing system 3010 at anaperture of the receiver.

In one or more of the various embodiments, the transmitter 3002 maytransmit a highly collimated beam in direction a. (While FIG. 30illustrates the system in two dimensions, showing a plane in which thetriangle ABC lies, another plane exists for each elevation angle ε.) Insome of the various embodiments, the transmit and receive (T_(x)-R_(x))system 3000 may be an epipolar aligned system. In some embodiments, fastscanning axis motion of the transmitter 3002 may scan a series of pointsC in rapid succession. In some embodiments, the transmit and receive(T_(x)-R_(x)) system 3000 may image reflections of these points as asuccession of imaged reflection points C′ in the sensor 3018, along thesame row. (Small deviations may occur due to factors such as non-idealmotions of a mirror and distortions of receiving optics.)

In one or more of the various embodiments, a row in the sensor may beformed by a triangle with height h=f and base corners C′ (e.g., thetriangle may be similar in shape as triangle ABC). In some of thevarious embodiments, the base length is d, which may be scaled down. Insome embodiments, the base corners may be α and β (e.g., the same as thebase corners of triangle ABC). In some embodiments, the transmit andreceive (T_(x)-R_(x)) system 3000 may determine that, based on the lawof similar triangles, d/f=D/Z. For example, the transmit and receive(T_(x)-R_(x)) system 3000 may know that, based on the law of similartriangles, if Z is 300 feet, D is 3 feet, and f is 10 mm, then d is1/100 of 10 mm (100 microns). In a 2k sensor with 1 micron pixels, dmeasured in pixels may be 100 pixels. In some of the variousembodiments, the transmit and receive (T_(x)-R_(x)) system 3000 maydetermine that the photons that reached the sensor at position C′traveled for approximately 600 ns (e.g., 300 ns from the transmitter3002 to target and 300 ns again back to receiver 3004). For a closertarget (e.g., at 200 feet), the photons may arrive at 400 ns and with asignificantly greater disparity (d) of 150 microns (e.g., 150 pixels).For 100 feet, the values may be 200 ns, 300 microns, and 300 pixels(e.g., as illustrated by table 1).

In one or more of the various embodiments, there may be a relationshipbetween where (e.g., which pixel, at how much disparity, or the like)and when the photons arrive. In some of the various embodiments, thefurther photons go, the longer a time period that the photons travel andthe smaller the disparity as explained above. In addition, according tothe inverse square law, a signal may attenuate proportionally totime-of-flight (e.g., a further distance results in fewer photons thatreturn to the aperture 3010).

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3000 may employ this type of ultra-fast pixelshuttering to individual pixels to open individual pixels “just-in-time”for capturing a signal. In some of the various embodiments, the transmitand receive (T_(x)-R_(x)) system 3000 may reduce ambient light “leakage”to an absolute (or practically absolute) minimum. For example, thetransmit and receive (T_(x)-R_(x)) system 3000 may open a correct pixel(e.g., selectively enabling a column and a row) for as little as 10nanoseconds, thereby capturing one hundred millionth (10⁻⁸) of afraction of the ambient light, even before filtering (e.g., with Braggtype narrow band pass filters).

In one or more of the various embodiments, the exemplary system may notrequire special narrow band pass filters. In some of the variousembodiments, ranging function of the transmit and receive (T_(x)-R_(x))system 3000 can be added to otherwise ordinary cameras. For example,rolling shutter camera with four-transistor pinned photodiodes may bemodified by adding a fast column gating function and an asynchronousbinary detection mode (e.g., as a “twitchy pixel” circuit as describedherein. In some embodiments, twitchy pixels may asynchronously transmitphotodiode rush current with minimal (e.g., nanosecond) latency to acolumn sense line. In some of the various embodiments, forthree-dimensional ranging, existing color pixels may be configured tosense visible color-coded (RG&B) tracer bullets. In some embodiments,the pixels may be configured to sequentially activate column decodercircuit sense lines to record received pulses. Alternatively, in someembodiments, in a two-dimensional mode, a full frame time integrationshutter exposure may be supported (e.g., with each successive row ofpixels being decoded in parallel).

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3000 may employ color codes to add contrast toenhance stereo decoding. In some of the various embodiments, thetransmit and receive (T_(x)-R_(x)) system 3000 may employ fast colorswitching (e.g., “chameleon mode”) to prevent ambiguity when intime-of-flight three-dimensional mode, thereby supporting, for example,three-times higher color voxel detection rates (e.g., FIG. 40).

In one or more of the various embodiments, the system 3000 mayselectively and momentarily increase sensitivity of each pixel. Forexample, the transmit and receive (T_(x)-R_(x)) system 3000 may increasea gain of a decoder circuit that a given pixel connects to. As anotherexample, the transmit and receive (T_(x)-R_(x)) system 3000 may increasegain for pixels in positions that the transmit and receive (T_(x)-R_(x))system 3000 determines are about to receive light that is mostattenuated (e.g., in comparison to light that one or more other pixelsare about to receive). In some of the various embodiments, the transmitand receive (T_(x)-R_(x)) system may determine that pixels that receiverays with lowest stereo disparity (e.g., in comparison to stereodisparity of rays that other pixels receive) receive photons that havetraveled furthest. In some embodiments, the transmit and receive(T_(x)-R_(x)) system 3000 may amplify signals from those pixels the mostin response to this determination to compensate for attenuation due tothe distance.

In one or more of the various embodiments, the sensor may have extrasensitive photodiodes (e.g., one or more of Avalanche Photo Diodes(APDs) or Single Photon Avalanche Diodes (SPADs)). In some of thevarious embodiments, these diodes may be reverse biased. In someembodiments, when triggered by photons, these diodes may avalanche,generating a strong current, and may then be quenched. In some of thevarious embodiments, the transmit and receive (T_(x)-R_(x)) system 3000may manage the bias of individual pixels in real-time according toselected sensitivity and gain. In some embodiments, the transmit andreceive (T_(x)-R_(x)) system 3000 may vary this bias from pixel-to-pixelover a period less than a microsecond.

In one or more of the various embodiments, a pixel may start out below acritical bias (e.g., keeping bias below APD linear mode). Just beforearrival of a first tracer bullet, the transmit and receive (T_(x)-R_(x))system 3000 may raise the bias to APD linear mode (e.g., close to butbelow a breakdown voltage) to detect relatively near reflections (e.g.,initially for first pixels to be activated, such as, for example, pixelsclose to a present d_(max) position). In some of the variousembodiments, the system 3000 may continue to rapidly increase the biasfor pixel positions closer to a d_(min) position (than to the d_(max)position) where, for example, even more sensitivity may improve rangedetection. In some embodiments, the transmit and receive (T_(x)-R_(x))system 3000 may temporarily raise the bias to Geiger mode. In some ofthe various embodiments, in Geiger mode (e.g., with reverse bias on aphotodiode set above a breakdown regime), the pixel becomes highlyunstable. In some embodiments, the transmit and receive (T_(x)-R_(x))system 3000 may limit operation in the Geiger mode to minimal (orpractically minimal) time intervals. For example, the transmit andreceive (T_(x)-R_(x)) system 3000 may place a few pixels in Geiger modeat one time and for brief periods (e.g., a hundred or so nanoseconds).As another example, the transmit and receive (T_(x)-R_(x)) system 3000may make pixel bias a function of both time and position. In someembodiments, the transmit and receive (T_(x)-R_(x)) system 3000 mayutilize the Geiger mode to one or more of minimize false detectionscaused by dark current, increase sensitivity of the sensor, increase adetection range, or avoid interference of ambient light.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3000 may rapidly vary responsivity of individualpixels to correspond to a certain expected time of flight at each pixel.For example, the transmit and receive (T_(x)-R_(x)) system 3000 maychange a photodiode's biasing voltage, and/or selectively enable a pixelto activate the pixel just in time and with an appropriate sensitivityfor an expected attenuation.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3000 may include a rapidly scanning laser beamsystem. In some of the various embodiments, for a plurality oftransmitting scan directions (e.g., tracer bullets fired in multipledirections a), pixels in a receiver 3004 may capture photons that arrivefrom various distances with different incoming directions β at differenttimes. In some embodiments, the receiver 3004 may first capture thosephotons that traversed the shortest distance. The receiver 3004 may nextcapture photons that traversed greater distances. In some of the variousembodiments, the transmit and receive (T_(x)-R_(x)) system 3000 mayeffectively range gate signals to respond only to light returning aftera certain delay. In some embodiments, the transmit and receive(T_(x)-R_(x)) system 3000 may do so to filter out responses that areoutside a desired range (e.g., one or more of too close or too far). Forexample, the transmit and receive (T_(x)-R_(x)) system 3000 may rapidlyand dynamically change a bias of a pixel's photodiode circuit.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3000 may employ a range selection function (e.g., aspatial range selection). In some of the various embodiments, thetransmit and receive (T_(x)-R_(x)) system 3000 may employ the rangeselection function for three-dimensional tracking where, for example,reflections from objects in a certain range (e.g., range Z_(min) toZ_(max)) are recorded.

FIG. 31 shows an exemplary transmit and receive system (T_(x)-R_(x))system 3100. For example, the (T_(x)-R_(x)) system 3100 may be the sameas or similar to one or more of those explained above. The (T_(x)-R_(x))system 3000 may include first and second receivers 3102 and 3104 (FIG.31 shows the second receiver 3104 in greater detail) and a transmitter3106. The second receiver 3104 may include an aperture 3108 and a sensor3110.

In one or more of the various embodiments, the transmitter 3106transmits a pulse at to in direction α₁. In some of the variousembodiments, the pulse might reflect at a first point 3112 at time t₁.In some embodiments, the reflection may arrive at a first pixel 3116 inthe sensor 3110 at time t₁ ⁺. Alternatively, the same pulse mightreflect at a second point 3118 at time t₂ and result in a second pixel3122 capturing the reflection at time t₂ ⁺. In some of the variousembodiments, the transmit and receive (T_(x)-R_(x)) system 3100 maydetermine that a difference in t_(ToF) is Δt=ΔZ (or ΔR)/2c. In one ormore of the various embodiments, at some time t₃, the transmitter 3106may transmit another pulse in a scan direction α₂. The other pulse mightreflect at a third point 3124 at time t₄. In some of the variousembodiments, the reflection may arrive at a third pixel 3126 in thesensor 3110 at time t₄ ⁺. Alternatively, the same pulse might reflect ata fourth point 3128 at time is and result in a fourth pixel capturingthe reflection at time t₅ ⁺. In some embodiments, the transmit andreceive (T_(x)-R_(x)) system 3100 may determine that the difference int_(ToF) is again Δt=ΔZ (or ΔR)/2c. As shown in FIG. 31, by sequentiallyactivating pixels at separate moments within these ranges in the sensor3110, the transmit and receive (T_(x)-R_(x)) system may provide a rangeof detection of ΔZ (or ΔR) (e.g., from an arc on which first and thirdpoints 3112 and 3124 reside to a further arc 3120).

FIG. 32 illustrates an exemplary transmit and receive (T_(x)-R_(x))system 3200 that may be an exemplary three-dimensional tracking system.For example, the transmit and receive (T_(x)-R_(x)) system 3200 may bethe same as or similar to one or more of those explained above. In oneor more of the various embodiments, the three-dimensional trackingsystem 3200 may “foveate” on a volumetric three-dimensional subsectionof a field-of-view space (e.g., as shown by the three-dimensional rangeselection, three-axis volumetric space selection). For example, thethree-dimensional tracking system 3200 may cause a scanning emitter 3206to illuminate certain ranges in horizontal (Δα) and vertical (Δε)directions. A stereo pair of first and second receivers 3202 and 3204may receive reflections of the illumination. As another example, thethree-dimensional tracking system may reach a subset of rows (ΔY) andcolumns (ΔX) in the second receiver 3204. As a further example, thethree-dimensional tracking system 3204 may select a detection range ΔZ.In some of the various embodiments, the three-dimensional trackingsystem 3204 may select the detection range by sequentially activatingnarrow sub-bands of pixels within those rows that the three-dimensionaltracking system reached.

FIG. 33 shows an exemplary transmit and receive (T_(x)-R_(x)) system3300 that manages exemplary pixels. For example, the transmit andreceive (T_(x)-R_(x)) system 3300 may be the same as or similar to oneor more of those explained above. The transmit and receive (T_(x)-R_(x))system 3300 may include a transmitter 3302 and a receiver 3304. Thereceiver 3304 may include a sensor that receives reflections via anaperture 3308. In some of the various embodiments, the sensor 3306 mayinclude one or more rows of pixels. For example, FIG. 33 shows a row, ofpixels in the sensor 3306.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3300 may bias and set a range of adjacent pixelsalong the row of pixels to open (e.g., shutter) in a direction ofdecreasing disparity. In some of the various embodiments, the transmitand receive (T_(x)-R_(x)) system 3300 may sequentially activatesub-pixels with increasingly sensitive detection (e.g., ramping gain)circuitry. In some embodiments, the transmit and receive (T_(x)-R_(x))system 3300 may initiate a sequential activation by activating a pixel3310 to detect an object 3316 at a Z_(min) range 3314. In someembodiments, the transmit and receive (T_(x)-R_(x)) system 3300 mayterminate the sequential activation after sequentially activating pixelsfrom the pixel 3310 to a pixel 3312 to detect an object 3320 in responseto reaching a Zmax range 3318. In some embodiments, the transmit andreceive (T_(x)-R_(x)) system 3300 may terminate the sequentialactivation in response to the transmit and receive (T_(x)-R_(x)) system3300 determining that reflections have ceased being detected. In someembodiments, the transmit and receive (T_(x)-R_(x)) system 3300 may timeout.

In some embodiments, the transmit and receive (T_(x)-R_(x)) system 3300may repeat the above process for a next tracer bullet in response totermination of the sequential activation. For example, the transmit andreceive (T_(x)-R_(x)) system 3300 may transmit the next tracer bullet(e.g., adjacent new a, next scan line position) that may cause the nexttracer bullet to arrive at the row of pixels when the above processrepeats. In some embodiments, the transmit and receive (T_(x)-R_(x))system 3300 may transmit the next tracer bullet at a time that may causethe next tracer bullet to already be “on its way” (e.g., half of time offlight to travel to target and another half time of flight on its wayback) when the transmit and receive (T_(x)-R_(x)) system 3300 capturesthe prior tracer bullet. In one or more of the various embodiments, thetransmit and receive (T_(x)-R_(x)) system 3300 may transmit the nexttracer bullet at a time that may cause the next tracer bullet to beseparable with regard to the prior tracer bullet from a perspective ofthe transmit and receive (T_(x)-R_(x)) system 3300 (e.g., unambiguouslydetectable).

In one or more of the various embodiments, at a certain time ti, firstphotons may arrive at row, in the sensor 3306 after having reflectedfrom a surface 3316 at a minimum range Z_(min) 3314. These “early birds”may be photons transmitted recently at t₀ in a certain transmissiondirection α_(i). A significant portion of them may return with a maximumdisparity d_(max). In some of the various embodiments, at asignificantly later time (e.g., up to a microsecond later), t_(j)(Δt=t_(j)-t_(i)=Δ_(ToF)=2*ΔZ/c, a time of flight of an additional roundtrip from point A to B and to C as the arrow of FIG. 33 indicates),final photons from the same transmission (in direction α_(i) andtransmitted at the same time to) may arrive at an entirely differentposition in the same row, in the sensor 3306. These “stragglers” may besignificantly fewer in proportion to the transmission (e.g., as comparedto that proportion for the early birds) and may arrive at a positionwith smallest disparity (e.g., shifted significantly away from the earlyarrivals). In some embodiments, arrival time, disparity, and photonfraction may be equally correlated measures of Z distance.

In some of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3300 may transmit the next tracer bullet at a timethat may cause the next tracer bullet's early birds from spuriousforeground objects to arrive subsequent to the transmit and receive(T_(x)-R_(x)) system's 3300 accounting for the prior tracer bullet'slast stragglers (e.g., the transmit and receive (T_(x)-R_(x)) system3300 may determine a time of flight for the prior tracer bullet's earlybirds and/or may determine a time difference in arrival of the priortracer bullet's early birds and the prior tracer bullet's laststragglers, and the transmit and receive (T_(x)-R_(x)) system 3300 maydelay transmitting the next tracer bullet based on one or more of thesedetermined time periods).

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3300 may reduce laser power for a tracer bullet toprevent stragglers. In some of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 3300 may disparity filter (e.g., block)early birds from very close foreground objects (e.g., spuriousforeground objects). In some embodiments, the transmit and receive(T_(x)-R_(x)) system 3300 may detect shadows (e.g., missing voxels) inresponse to partial overlaps between one or more very close foregroundobjects (e.g., spurious foreground objects) and a foreground targetobject. For example, the transmit and receive (T_(x)-R_(x)) system 3300may, while locked onto and ranging the foreground target object, detecta lower “hit” rate with regard to reflections from the foreground targetobjects (e.g., due to flack, snow, rain, or fog).

FIG. 34 illustrates an exemplary transmit and receive (T_(x)-R_(x))system 3400. For example, the transmit and receive (T_(x)-R_(x)) system3400 may be the same as or similar to one or more of those explainedabove. The transmit and receive (T_(x)-R_(x)) system 3400 may include atransmitter 3402 and a receiver 3404.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3400 may detect a foreground object 3406 against abackground 3408 at a first pixel 3412. In some of the variousembodiments, the transmit and receive (T_(x)-R_(x)) system 3400 may“remove” the background 3408. In some embodiments, the transmit andreceive (T_(x)-R_(x)) system 3400 may remove the background 3408byselecting a moderate (but sufficient) intensity for a scanning beam. Insome embodiments, the transmit and receive (T_(x)-R_(x)) system 3400 mayremove the background 3408 by selecting an appropriate threshold (e.g.,a floor) for a receive sensitivity of the receive system 3404 of thetransmit and receive (T_(x)-R_(x)) system 3400. In some embodiments, thetransmit and receive (T_(x)-R_(x)) system 3400 may remove the background3408by “turning off” pixels after a short time-of-flight interval sothat light that arrives after this interval may not trigger a detector(e.g., a “twitchy pixel”, APD, or SPAD) at a lesser disparity (e.g.,pixel locations) beyond the range drain. For example, light thatreflects off a surface at a distance further than a threshold distance3410 may arrive at the sensor at one or more pixels in a first direction3418 of a second pixel 3420. The transmit and receive (T_(x)-R_(x))system 3400 may not activate pixels that reside in the first direction3418 of the second pixel 3420 or may ignore signals from those pixels.

In one or more of the various embodiments, small spurious foregroundobjects 3416 may interfere with an ability of the transmit and receive(T_(x)-R_(x)) system 3400 to detect the foreground object 3406 as shownin FIG. 34. In some of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3400 may ignore reflections of the scanning beamfrom the small spurious foreground objects 3416 (e.g., the transmit andreceive (T_(x)-R_(x)) system 3400 may eliminate spurious foregroundlight). In some embodiments, the transmit and receive (T_(x)-R_(x))system 3400 may ignore (e.g., reject) early arriving photons (“gatecrashers”) that reflect off surfaces that are closer than a seconddistance 3414. In some embodiments, the transmit and receive(T_(x)-R_(x)) 3400 system may ignore reflections of the scanning beamfrom the small spurious foreground objects 3416 by delaying enablement“firing” by pixels until after a certain minimal latency. For example,after the beam transmits in direction cu, the transmit and receive(T_(x)-R_(x)) system 3400 may delay releasing (e.g., biasing) the pixelsuntil after a minimum time-of-flight interval. In some embodiments, thetransmit and receive (T_(x)-R_(x)) system 3400 may select a minimumtime-of-flight interval that is more than twice a travel time in onedirection (R_(min)/c). For example, the transmit and receive(T_(x)-R_(x)) system 3400 may prevent activation of pixels that residein a second direction 3424 of a pixel that would receive a reflectionfrom any object at the second distance 3414 or may ignore signals fromthose pixels.

In one or more of the various embodiments, the foreground objects 3416may partially occlude the background, but the transmit and receive(T_(x)-R_(x)) system 3400 may avoid recording foreground image voxels.The above-explained three-dimensional voxel filter may be useful in, forexample, rejecting spurious reflections caused by fog, rain, or snow.The above-explained three-dimensional voxel filter may mitigate blindinga sensor of the receive system 3404 and/or overloading one or more ofbusses, logic, or VPU (Visual Processing Unit) of the transmit andreceive (T_(x)-R_(x)) system 3400 with such spurious noisy voxel data.

FIG. 35 shows an exemplary transmit and receive (T_(x)-R_(x)) system3500 that scans an exemplary relatively clear sky for exemplaryprojectiles. For example, the transmit and receive (T_(x)-R_(x)) system3500 may be the same as or similar to one or more of those explainedabove. In one or more of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 3500 may include a transmitter 3502 and areceiver 3504. In some of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 3500 may Z-lock onto a surface to bemapped. In some of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3500 may Z-lock onto a surface to be mapped in highresolution or for robust tracking of a high-speed moving object 3506.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3500 may time-gate a subset of pixels in a sensor3524 of the receive system 3504. In some of the various embodiments, thetransmit and receive (T_(x)-R_(x)) system 3500 may time-gate the subsetof pixels to be open for a limited range 3508 of Z (e.g., for a 500 ft.Z-range). In some embodiments, the transmit and receive (T_(x)-R_(x))system 3500 may select the Z-range 3508 for scanning a volume of skyabove a suspected missile launch site to be monitored (e.g., at 5000feet distance where the range may be 10% of the total distance).

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3500 may, during each scan, set each pixel in thesensor 3524 to open for a given time period (e.g., a few nanoseconds) ata sufficient range of pixel disparities (e.g., from a first pixel 3526that corresponds to a far range 3512 through a second pixel 3528 thatcorresponds to a near range 3510) and over a sufficient time-of-flightrange. In some of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3500 may confine detection to a trail of nanosecondsequentially shuttered active pixels, following an image of the scanbeam trajectory in the sensor 3524. In some embodiments, the transmitand receive (T_(x)-R_(x)) system 3500 may (e.g., when the transmit andreceive (T_(x)-R_(x)) system 3500 activates the pixels) set asensitivity of the pixels to an appropriate gain for the selected range3508.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3500 may detect the object 3506 in response to thefirst reflected photons arriving at a pixel 3530. In some of the variousembodiments, the transmit and receive (T_(x)-R_(x)) system 3500 may(e.g., in detecting the object 3506) precisely observe disparity (e.g.,pixel shift di) and time (e.g., ToF_(i) delay) for the first reflectedphotons. In some embodiments, the transmit and receive (T_(x)-R_(x))system 3500 may instantly (or practically instantly) calculate an exact(or practically exact) range. In some embodiments, the transmit andreceive (T_(x)-R_(x)) system 3500 may adjust one or more of sensitivity,disparity (e.g., by reducing a range of utilized pixels to those from athird pixel 3532 that corresponds to a first middle distance 3520 to afourth pixel 3534 that corresponds to a second middle distance 3518), ortime-of-flight ranges based on that detected position. For example, thetransmit and receive (T_(x)-R_(x)) system 3500 may optimize one or moreof the sensitivity, disparity, or time-of-flight ranges to a minimalenvelope 3516 around that detected position. In some embodiments, thetransmit and receive (T_(x)-R_(x)) system 3500 may set the scan beam toa corrected power level, may set a gain of the sensor, may set thedisparity, and may set pixel timing (e.g., each based on that detectedposition). In some embodiments, the transmit and receive (T_(x)-R_(x))system 3500 may, given repeated successful detections, predict (e.g.,estimate) a trajectory 3522 more and more accurately. In someembodiments, the transmit and receive (T_(x)-R_(x)) system 3500 maytrack and increasingly lock onto the missile. In one or more of thevarious embodiments, the transmit and receive (T_(x)-R_(x)) system 3500may employ one or more portions of this method help ignore chaff orother objects (O_(sf)) 3514 that a party may deliberately launched toconfuse conventional RADAR or LIDAR systems. In some of the variousembodiments, the transmit and receive (T_(x)-R_(x)) system 3500 mayemploy one or more portions of this method to better track the objectswhen they are partially or temporally occluded by other objects.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3500 may, when the transmit and receive(T_(x)-R_(x)) system 3500 locks onto a surface, set a sharper, narroweracceptance margin for each successive (e.g., adjacent in time and space)pixel set to detect the next tracer bullet reflection. In some of thevarious embodiments, the transmit and receive (T_(x)-R_(x)) system 3500may reduce an active window of acceptable time-of-flight and disparityranges. For example, the transmit and receive (T_(x)-R_(x)) system 3500may focus on detecting objects of interest that have dimensions of 10feet or less. In such an example, the transmit and receive (T_(x)-R_(x))system 3500 may set a 20 ns time-of-flight margin and a few pixels ofdisparity around predicted next locations.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3500 may, in making this prediction, considerrelative motion between an observer and the tracked object (e.g., byKalman filtering techniques predicting the object's three-dimensionalmotion trajectory). In some of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 3500 may, when the transmit and receive(T_(x)-R_(x)) system 3500 has acquired such a surface, automatically“unlock” when locked tracking has failed. For example, the transmit andreceive (T_(x)-R_(x)) system 3500 may, when the transmit and receive(T_(x)-R_(x)) system 3500 has acquired such a surface, automatically“unlock” after the scan fails to detect adjacent voxels (e.g., if thetransmit and receive (T_(x)-R_(x)) system 3500 fails to detect tracerbullet reflections inside a narrower window).

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3500 may, by employing one or more portions of theabove-explained method, lock onto multiple objects at different Z-rangesand/or at different positions in the field of view. In some of thevarious embodiments, the transmit and receive (T_(x)-R_(x)) system 3500may, by employing one or more portions of the above-explained method,lock onto multiple objects at different Z-ranges and/or at differentpositions in the field of view within the same scan. In someembodiments, the transmit and receive (T_(x)-R_(x)) system 3500 may, byemploying one or more portions of the above-explained method, lock ontomultiple objects at different Z-ranges and/or at different positions inthe field of view with pixels in a sensor line individually preset toeach of a plurality of narrower lock-on ranges (e.g. as shown in one ormore of FIGS. 39 or 40). In one or more of the various embodiments, thetransmit and receive (T_(x)-R_(x)) system 3500 may reset to a widersearch range between objects to detect new, previously undetectedobjects.

FIG. 36 illustrates an exemplary transmit and receive (T_(x)-R_(x))system 3600 that may be an exemplary fast pixel sequential scanningLIDAR system. For example, the transmit and receive (T_(x)-R_(x)) system3600 may be the same as or similar to one or more of those explainedabove. In one or more of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 3600 may include a transmit system 3602 anda receive system 3604. In some of the various embodiments, the transmitsystem 3602 may be arranged at a position that may be close to aposition of the receive system 3604. In some embodiments, the transmitsystem 3602 and the receive system 3604 may be positioned on the sameaxis as each other. In some embodiments, the transmit system 3602 andthe receive system 3604 may be optically combined.

In one or more of the various embodiments, the transmit system 3602 mayfire at time to (departure) in direction a. In some of the variousembodiments, the receive system 3604 may, after time of flight (e.g., xnanoseconds) subsequent to time t_(d), receive a reflected signal off anobject 3608 at t_(a) (arrival). In some embodiments, the receive system3604 may detect the reflected signal as having come from direction β. Inone or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3600 may estimate a duration that photons of thereflected signal were in flight. In some of the various embodiments, thetransmit and receive (T_(x)-R_(x)) system 3600 may make this estimationbased on presumptions that D is zero (or nearly zero) and that α isequal (or approximately equal) to β. For example, the transmit andreceive (T_(x)-R_(x)) system 3600 may make this estimation based on adetermined place (e.g., column) that the photons arrived and adetermined time of arrival for the photons in the sensor. In one or moreof the various embodiments, the transmit and receive (T_(x)-R_(x))system 3600 may know, for each incoming direction β, the departure timein the corresponding direction α(=β) during the most recent scan. Insome embodiments, the transmit and receive (T_(x)-R_(x)) system 3600 maybe fast enough to employ a continuously illuminated laser scanning beambecause each instantaneous transmit direction α may be determined afterdetection of a reflection (for example, by correlating the direction αwith an observed return direction β of the detected reflection). In someof the various embodiments, the receive system 3604 may record thearrival time and the time of flight. In some embodiments, the transmitand receive (T_(x)-R_(x)) system 3600 may calculate a range R based onR=c(t_(a)-t_(d))/2.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3600 may employ fast scanning (e.g., 50 kHz). Insome of the various embodiments, the transmit and receive (T_(x)-R_(x))system 3600 may, while employing the fast scanning, correlate every100^(th) of a degree with a nanosecond. For example, the transmit andreceive (T_(x)-R_(x)) system 3600 may use “twitchy pixel” or SPARreceiver with 10,000 columns to do so.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3600 may include a single line (e.g., linearsensor) receiver. In some of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 3600 may include a tilting mirror thatrotates the scan in a vertical direction. In some embodiments, an“anti-epsilon mirror” of the receive system 3604 may match an epsilon(e.g., elevation) mirror of the transmit system 3602. In someembodiments, the transmit and receive (T_(x)-R_(x)) system 3600 mayrange lock by reducing an active pixel sub-window during the scan.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3600 may use one or more of small range expectationor location determinism to range lock incoming photons. In some of thevarious embodiments, the transmit and receive (T_(x)-R_(x)) system 3600may use one or more of small range or location determinism to shuttersuccessive incoming photons. In some embodiments, the transmit andreceive (T_(x)-R_(x)) system 3600 may lock onto and range athree-dimensional surface with a particular volume.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3600 may utilize a lack of disparity (e.g., a lackof ambiguity due to “Z/d pixel overlap”). In some of the variousembodiments, the transmit and receive (T_(x)-R_(x)) system 3600 may mappixel positions in a row (e.g., column numbers) directly intofield-of-view positions. In some embodiments, the transmit and receive(T_(x)-R_(x)) system 3600 may provide fast scanning (e.g., via a 50 kHzresonant MEMS mirror). For example, the transmit and receive(T_(x)-R_(x)) system 3600 may provide scanning at one nanosecond pervoxel. In one or more of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 3600 may achieve such fast scanning byreducing the detection range to, for example, 10 feet (e.g., auto-lockon a surface). In some of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system3600 may, in response to reducing the range,simultaneously fire multiple column decoders. For example, the transmitand receive (T_(x)-R_(x)) system 3600 may drive the column decoders viatwitchy pixels, APDs, or SPADs.

FIG. 37 shows an exemplary transmit and receive (T_(x)-R_(x)) system3700 (e.g., an exemplary assisted stereo scanning system) with anexemplary fast sliding window logic epipolar parallel stereo decoder.For example, the transmit and receive (T_(x)-R_(x)) system 3700 may bethe same as or similar to one or more of those explained above. In oneor more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3700 may include a transmitter 3702, a firstreceiver 3704, and a second receiver 3706. In some of the variousembodiments, the two receivers 3704 and 3706 may be properly aligned ina pure rectified epipolar arrangement. In some embodiments, the tworeceivers 3704 and 3706 may compensate for lens effects (e.g., via theiralignment). In some embodiments, the transmit and receive (T_(x)-R_(x))system 3700 may include a sliding register comparator that may findstereo pair pixel correspondences (e.g., based on the alignment of thetwo receivers 3704 and 3706).

In one or more of the various embodiments, the transmit system 3702 mayproject DeBruijn color codes (e.g., 7 possible binary colors 2³-1:RGBCMY&W). FIG. 37 shows the color codes as decimal numbers 1-7 forclarity. In some of the various embodiments, an object may reflect thecode “14527”. For example, the “1” may reflect off a first portion 3708of the object. The “4” may reflect off a second portion 3710 of theobject. The “5” may reflect off a third portion 3712 of the object. The“2” may reflect off a fourth portion 3714 of the object. The “7” mayreflect off a fifth portion 3716 of the object. In some embodiments,left and right cameras of the two receivers 3704 and 3706 may see thesecodes in reverse (e.g., as “72541”) (assuming no occlusions). Forexample, a first pixel 3718 and a second pixel 3720 may receive thereflection of the “1”. A third pixel 3722 and a fourth pixel 3724 mayreceive the reflection of the “4”. A fifth pixel 3726 and a sixth pixel3728 may receive the reflection of the “5”. A seventh pixel 3730 and aneighth pixel 3732 may receive the reflection of the “2”. A ninth pixel3734 and a tenth pixel 3736 may receive the reflection of the “7”.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3700 may estimate a distance to a surface of theobject based on the color code. In some of the various embodiments, thetransmit and receive (T_(x)-R_(x)) system 3700 may estimate a disparityof codes (e.g., colors) that match between the left camera and the rightcamera. In some embodiments, the transmit and receive (T_(x)-R_(x))system 3700 may, due at least in part to the rectified epipolaralignment, look up a given code captured in the left camera with amatching code in the right camera and then determine a magnitude ofpixel disparity between the matched (e.g., identical) codes. Forexample, the left and right cameras may include 4K sensors with 1 micronpixels (e.g., 4 mm array width). As explained above with regard to FIG.30 (e.g., using the formula d/f=D/Z), the transmit and receive(T_(x)-R_(x)) system 3700 may determine that, if the disparity betweencodes is 100 pixels (e.g., 100 micron) and a focal length of the leftand right cameras is 10 mm, d/f=1/100. The transmit and receive(T_(x)-R_(x)) system 3700 may also determine that, if a baselinedistance between the left and right cameras is 3 feet, Z is 100×3feet=300 feet.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3700 may include an assisted stereo scanningsystem. The assisted stereo scanning system may, for example, projectfive sequential De Bruijn colors on a back of a truck in a fewnanoseconds. In some of the various embodiments, the left and rightcameras may each have rolling shutters. In some embodiments, the rollingshutters of the left camera may be synchronized with the rollingshutters of the right camera. In some embodiments, the transmit andreceive (T_(x)-R_(x)) system 3700 may include store the captured codesfor each of the left and right cameras.

For example, the transmit and receive (T_(x)-R_(x)) system 3700 mayinclude one or more hardware registers that store binary equivalents ofeach captured color (for simplicity, FIG. 37 shows decimals 1-7 insteadof 001, 010, 011, 100,101,110,111). In some embodiments, the transmitand receive (T_(x)-R_(x)) system 3700 may exclude a binary combinationof 000 (e.g., black). In one or more of the various embodiments, thetransmit and receive (T_(x)-R_(x)) system 3700 may, after storing aquantity of color codes that equates to an expected length of the codes(e.g., 20 microseconds after), reading the two synchronized rollingshutters. For example, the transmit and receive (T_(x)-R_(x)) system3700 may read two hardware registers 3738 and 3740 after (e.g., 20microseconds after) the two hardware registers fill up with the binaryequivalents of the captured codes. In one or more of the variousembodiments, the transmit and receive (T_(x)-R_(x)) system 3700 maycompare values of row, of the left camera to values of equivalent row,in the right camera.

In one or more of the various embodiments, the code (e.g., “72541”) mayoccur in a same row I of both of the left and right cameras. In some ofthe various embodiments, the transmit and receive (T_(x)-R_(x)) system3700 may load both of these left and right versions of the same row Iinto respective registers 3738 and 3740. In some embodiments, thetransmit and receive (T_(x)-R_(x)) system 3700 may load the rightversion of the row I into the respective register with an offset (e.g.,a pixel shift) of a particular amount (e.g., d_(max)). In one or more ofthe various embodiments, the transmit and receive (T_(x)-R_(x)) system3700 may subtract the register 3740 of the right camera from theregister 37338 of the left camera. In some of the various embodiments,the transmit and receive (T_(x)-R_(x)) system 3700 may determine matchesor, additionally or alternatively, “near-matches” (e.g., where thetransmit and receive (T_(x)-R_(x)) system 3700 determines that adifference for shifted positions falls below a certain threshold). Forexample, the transmit and receive (T_(x)-R_(x)) system 3700 maydetermine that a match (or near-match) exists when a comparison of thetwo registers 3738 and 3740 results in zeros (e.g., “00000”). In someembodiments, the transmit and receive (T_(x)-R_(x)) system 3700 mayrecord the determined matches (or near-matches) with a present pixelshift (e.g., disparity for those matches). In one or more of the variousembodiments, the transmit and receive (T_(x)-R_(x)) system 3700 may thenleft-shift values in the right register 3740 (e.g., toward lowerdisparity) and repeat the process. In some of the various embodiments,the transmit and receive (T_(x)-R_(x)) system 3700 may shift one or moreof the registers 3738 and 3740 multiple times prior to determiningwhether a match (or near-match) exists (e.g., to align the two “72541”codes as shown in FIG. 37).

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3700 may provide very fast stereo matching (e.g.,more than fast enough to support a 4K stereo decoder at up to 1 millionlines per second). In some of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 3738 and 3740 may include two 4K cameras(e.g., each with 4K by 2K pixels=8 million pixels each). In someembodiments, the transmit and receive (T_(x)-R_(x)) system 3738 and 3740may decode, for example, 250 k lines per second (e.g., providing 125frames at 4K resolution per second in full color=8M RBG voxels/frame, orrates up to 1 Giga voxels per second).

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3700 may, even though each row in the sensor maycontain, for example, 4000 pixel values, consider a certain subrange ofpotential disparities (e.g., d_(min) to d_(max)). For example, thetransmit and receive (T_(x)-R_(x)) system 3700 may have 40 degrees ofview with 4K pixels across the field of view (e.g., 100 pixels perdegree). In the case of this example, the transmit and receive(T_(x)-R_(x)) system 3700 may, for 1) a baseline offset of 3 feetbetween the left and right cameras, 2) a 10 mm focal length, and 3) adepth range of 200′ Z_(min) to 400′ Z_(max), consider range disparitiesof 150 pixels to 75 pixels (e.g., d_(max) and d_(min) respectively)(e.g., a net 75 pixels). In the case of this example, the transmit andreceive (T_(x)-R_(x)) system 3700 may shift the values in the hardwareregisters 75 times to find possible matches in this range. In one ormore of the various embodiments, the transmit and receive (T_(x)-R_(x))system 3700 may, where implementing the registers 3738 and 3740 inhardware, do so in 75 clocks (e.g., 75 ns with a 1 GHz system). In someof the various embodiments, the transmit and receive (T_(x)-R_(x))system 3700 may take less than a microsecond to find matches in thepixels. For example, the transmit and receive (T_(x)-R_(x)) system 3700may subtract the right register 3740 with the right sensor pixel valuesfrom the left register 3738 with the left sensor pixel values andincrement a lateral shift by one pixel 75 times. In some embodiments,the transmit and receive (T_(x)-R_(x)) system 3700 may start at maximumdisparity and work towards minimum disparity.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3700 may include one or more memories that storecode that, when one or more processors of the transmit and receive(T_(x)-R_(x)) system 3700 execute the code, perform the above process.An example of such code is as follows.

(STEP 0)

Load R_(xl) values row_(i) in to reg_(L) (4 k pixel_values)

Load R_(xr) values row_(i) in to reg_(R) (4 k pixel_values)

Set d to d=d_(max)

(STEP 1) Offset reg_(R) d pixels from reg_(L) (See, e.g., FIG. 37)

Subtract non-zero values from each other in each column

If difference is zero,

-   -   OR, optionally, if difference is below noise threshold.

Then for those pixels note the disparity value

-   -   AND, optionally, with a confidence value base on the difference

Decrement the disparity d by one pixel

And repeat step 1 until d=d_(min)

All pixels with non zero values should have a disparity value

-   -   (Optionally with a confidence value, based on the smallest        mismatch)

Repeat this procedure (Step 0) for each row.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3700 may read a row in one of the cameras in about10-20 microseconds. In some of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 3700 may take less than 1 microseconds(e.g., 1000 clocks @ 1 GHz) to run the above process in hardware. Insome embodiments, the transmit and receive (T_(x)-R_(x)) system 3700 maystream the two receiver outputs through a hardware block that extractsbest possible pixel matches in real time. For example, the transmit andreceive (T_(x)-R_(x)) system 3700 may stream the two receiver outputssequentially, row by row at up to 100 fps.

In some embodiments, the transmit and receive (T_(x)-R_(x)) system 3700may compute all of the Z values with minimal (or practically minimal)latency. In one or more of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 3700 may be a very fast assisted stereohybrid LIDAR system.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 3700 may include a pair of high resolution sensors,combined with a fast pixel sequential scanning system in an epipolararrangement as explained above. In some of the various embodiments, thetransmit and receive (T_(x)-R_(x)) system 3700 can detect, localize, andtrack, for example, as many as 1 billion color voxels per second. Insome embodiments, the system may triangulate two contrast enhanced videostreams in hardware. In some embodiments, the transmit and receive(T_(x)-R_(x)) system 3700 may, with scanning laser illumination, trackhigh-speed objects without blur and with great contrast. In someembodiments, the transmit and receive (T_(x)-R_(x)) system 3700 mayprovide high-resolution color three-dimensional images and highlyaccurate motion trajectories. In some embodiments, the transmit andreceive (T_(x)-R_(x)) system may, using tight control over gain, usingtight control over sensitivity, and using ultra-short activation ofindividual pixels, track while remaining highly robust to ambient light.

FIG. 38 shows an exemplary transmit and receive (T_(x)-R_(x)) system3800 that may sequentially set and activate exemplary pixels. Forexample, the transmit and receive (T_(x)-R_(x)) system 3800 may be thesame as or similar to one or more of those explained above. In one ormore of the various embodiments, the transmit and receive (T_(x)-R_(x))system 3800 may include a transmitter 3802 and a receiver 3804. Thereceiver 3804 may include a sensor 3806. The receiver 3804 may set again amplitude 3808 that may be different for different pixels of thesensor 3806. For example, the receiver 3804 may set the gains based on again envelope 3810. In one or more of the various embodiments, thereceiver 3804 may set a first gain amplitude 3812 for a first pixel3814, a second gain amplitude 3816 for a second pixel 3818, and a thirdgain amplitude 3820 for a third pixel 3822.

In one or more of the various embodiments, the receiver 3804 maysequentially set the gains 3812, 3816, and 3820 in the individual pixels3814, 3818, and 3822. In some of the various embodiments, the transmitand receive (T_(x)-R_(x)) system 3800 may sequentially set and activatedthe pixels 3814, 3818, and 3822 by setting the gains 3812, 3816, and3820 at times t₁, t₂, t₃ respectively. In some embodiments, the transmitand receive (T_(x)-R_(x)) system 3800 may employ smart system logic. Forexample, the transmit and receive (T_(x)-R_(x)) system 3800 may expect aphoton signal to arrive at one of the pixels 3814, 3818, and 3822 at agiven moment in time as explained above (e.g., a reflection from a firstposition 3824 may arrive prior to a reflection from a second position3826, which may arrive prior to a reflection from a third position3828). In one or more of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 3800 may activate successive pixels inaccordance with such expectation. In some of the various embodiments,the transmit and receive (T_(x)-R_(x)) system 3800 may compensate for anexpected attenuation (e.g., based on incremental Z values when steppingthrough each successive pixel). For example, the transmit and receive(T_(x)-R_(x)) system 3800 may apply a ramping gain function GR(t) todetermine the envelope 3810, ramping the gains 3812, 3816, and 3820rapidly in time as a function time (t_(ToF)).

FIG. 39 illustrates an exemplary transmit and receive (T_(x)-R_(x))system 3900 that may employ successive exemplary rays to obtainexemplary corresponding disparities. For example, the transmit andreceive (T_(x)-R_(x)) system 3900 may be the same as or similar to oneor more of those explained above. In one or more of the variousembodiments, the transmit and receive (T_(x)-R_(x)) system 3900 mayinclude a transmitter 3902 and a receiver 3904. In some of the variousembodiments, the transmitter 3902 may successively emit a first beam3906 at a first scan angle α₁, a second beam 3908 at a second scan angleα₂, and a third beam 3910 at a third scan angle α₃.

In one or more of the various embodiments, the first beam 3906 mayreflect from a first object 3912. A first pixel 3914 of the receiver3904 may capture the deflection of the first beam 3906. In someembodiments, the transmit and receive (T_(x)-R_(x)) system 3900 may seta small range of disparities Δd₁ (e.g., that corresponds to a depthrange 3916 from a first distance 3918 to a second distance 3920) aroundpixels in the sensor in the receiver 3904. The second beam 3908 maymiss, and the transmit and receive (T_(x)-R_(x)) system 3900 may fail tocapture a reflection of the second beam 3908. The third beam 3910 mayhit and deflect off a second object 3922. A second pixel 3924 of thereceiver 3904 may capture the reflection of the third beam 3910. In someembodiments, the transmit and receive (T_(x)-R_(x)) system 3900 may seta range of disparities Δd₂ (e.g., that corresponds to a depth range 3926from a third distance 3928 to a fourth distance 3930) around pixels inthe sensor.

FIG. 40 shows an exemplary transmit and receive (T_(x)-R_(x)) system4000 that may employ exemplary color coding (e.g., time-of-flight colorcoding) to prevent exemplary ambiguity and/or to increase exemplaryscanning rates (e.g., time-of-flight scanning rates). For example, thetransmit and receive (T_(x)-R_(x)) system may be the same as or similarto one or more of those explained above. In one or more of the variousembodiments, the transmit and receive (T_(x)-R_(x)) system 4000 mayinclude a transmitter 4002 and a receiver 4004.

In one or more of the various embodiments, the transmitter 4002 maylaunch successive tracer bullets within a sizeable range from a firstdistance 4006 to a second distance 4008 (e.g., Z₁ to Z₂) and indifferent directions α (e.g., α₁ and α₂) at different times. In some ofthe various embodiments, the tracer bullets may return to the transmitand receive (T_(x)-R_(x)) system 4000 out of order. For example, areflection at point 4012 may arrive at pixel 4014 at a first time, areflection at point 4016 may arrive at a pixel 4018 at a second time, areflection at point 4020 may arrive at a pixel 4022 at a third time, anda reflection at point 4024 may arrive at a pixel 4026 at a fourth time(e.g., where the first through fourth times are in successive temporalorder—the pixels may capture reflections from further distancessubsequent to capturing reflections of closer distances). In someembodiments, the transmit and receive (T_(x)-R_(x)) system 4000 mayexperience a certain degree of ambiguity (e.g., a scanner of thetransmit and receive (T_(x)-R_(x)) system 4000 has avoided movement in aparticular direction by as great of an increment as shown here, forexample Δα). For example, these reflections from further points mayoverlap (e.g., intermingle) in the sensor 4010 (e.g., both temporally,such as, for example, the third time may occur before the second time,and spatially). In one or more of the various embodiments, the transmitand receive (T_(x)-R_(x)) system 4000 may provide smaller angularincrements (e.g., smaller than those shown) without causing ambiguity.For example, the transmit and receive (T_(x)-R_(x)) system 4000 may doso for high speed, high resolution scanning of voxels. In some of thevarious embodiments, the transmit and receive (T_(x)-R_(x)) system 4000may change a scanning beam signal. In some embodiments, the transmit andreceive (T_(x)-R_(x)) system 4000 may rapidly change a color of thescanning beam in real time. In one or more of the various embodiments,the transmit and receive (T_(x)-R_(x)) system 4000 may, after a smallerscan mirror rotation increment (e.g., Δα/2), fire tracer bullets thatreflect off points 4028 (which directs a reflection to arrive at pixel4030 at a fifth time) and 4032 (which directs a reflection to arrive atpixel 4034 at a sixth time). In some of the various embodiments, thetransmit and receive (T_(x)-R_(x)) system 4000 may change the color ofthe scanning beam for each successive tracer bullet (e.g., from a bluetracer bullet to a red tracer bullet). In some embodiments, the transmitand receive (T_(x)-R_(x)) system 4000 may utilize the different colorsof each successive tracer bullet to avoid ambiguity in detection. In oneor more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4000 may detect the reflected light that arrives atthe fifth and sixth times via red pixels (e.g., instead of via bluepixels). For example, as explained above, the transmit and receive(T_(x)-R_(x)) system 4000 may include RGB hybrid sensors (e.g.,three-dimensional and/or two-dimensional hybrid sensors). In some of thevarious embodiments, the transmit and receive (T_(x)-R_(x)) system 4000may switch one or more other light attributes (e.g., polarization,intensity, phase) and/or us simultaneously mixed colors. In someembodiments, the transmit and receive (T_(x)-R_(x)) system 4000 mayswitch one or more light attributes in response to determining thatambiguity exceeds a threshold (e.g., as determined by logic of thetransmit and receive (T_(x)-R_(x)) system 4000 or by a CPU-based controlsystem that monitors the transmit and receive (T_(x)-R_(x)) system4000). As shown above, the transmit and receive (T_(x)-R_(x)) system4000 may reduce ambiguity and/or provide greater detection speeds (e.g.,greater voxel ranges) by employing one or more portions of this process.

Additional Illustrated Circuitry of the Sensing Systems

FIG. 41 illustrates an exemplary sensor portion 4100 that includes anexemplary “twitchy pixel” 4102. An exemplary sensor of an exemplarytransmit and receive (T_(x)-R_(x)) system may include the sensor portion4100. For example, the transmit and receive (T_(x)-R_(x)) system may bethe same as or similar to one or more of those explained above. In oneor more of the various embodiments, the twitchy pixel 4102 may employreal-time pixel shuttering. In some of the various embodiments, thetwitchy pixel 4102 may employ real-time column shuttering. In someembodiments, the twitchy pixel 4102 may employ real-time row shuttering.In some of the various embodiments, the twitchy pixel 4102 may employdynamic sensitivity adjustment.

In one or more of the various embodiments, the twitchy pixel 4102 mayinclude a pinned photodiode 4104. The PDD 4104 may capture a reflectionof a scanning spot 4106 at t₀. In some of the various embodiments,photon current may rush via a transfer gate 4108 (e.g., at t₁). In someembodiments, an active row control line 4110 may activate the transfergate 4108. For example, the active row control line 4110 may activateindividual pixels in a first row 4112 of the sensor.

In one or more of the various embodiments, the rush current may thenreach an input of an amplification circuit 4114 (or source follower). Insome of the various embodiments, this input may be weakly tied to V_(DD)with a resistor R. The weak pull-up resistor R may hold the input of theamplification circuit 4114 at V_(DD) until the rush current temporarilypulls the input low. In some embodiments, this sudden “twitch” may beamplified and transmitted to a column sense line 4116 at t₂. In some ofthe various embodiments, the amplified “twitch” signal may betransmitted to a row sense line. Alternatively, the amplified “twitch”signal may be simultaneously transmitted to both row and column senselines so that both the vertical and horizontal positions of the“twitched” pixel are instantaneously transmitted to a detection system.

In one or more of the various embodiments, the amplified transmissionmay arrive at a transistor 4118 of a column decoder/amplifier 4120 att₃. In some of the various embodiments, an active column control line4122 may activate the transistor 4118 of the column decoder/amplifierCDA. In some embodiments, the column decoder/amplifier 4120 may amplifythe transmission again at t₄. In one or more of the various embodiments,the column decoder/amplifier 4120 may transmit the again amplifiedtransmission at time is (e.g., transmit an amplified detection signalover an output line 4124 to a remote processing system). In someembodiments, the output line 4124 may provide outputs for each pixel ina column 4126.

FIG. 42 shows exemplary activation and gain control circuitry 4200 of anexemplary pixel 4204. For example, the pixel 4204 may be the same as orsimilar to one or more of those explained above. In one or more of thevarious embodiments, the activation and gain control circuitry 4200 maybe controlled by a column pixel gain control line 4206. In some of thevarious embodiments, the column pixel gain control line 4206, inaddition to a row select line 4208, may ensure that activation andsensitivity of individual pixels can be changed in real time (e.g., viaand/or during one or more of the above-explained processes). Forexample, one or more signals transmitted over the column pixel gaincontrol line 4206 may cause a gain control circuit 4210 to vary anamount of amplification applied to signals that the pixel 4204 outputsto a column sense line 4212.

FIG. 43 illustrates an exemplary activation and gain control circuitry4300 of an exemplary pixel 4302. For example, the pixel 4302 may be thesame as or similar to one or more of those explained above. In one ormore of the various embodiments, the gain control circuitry 4300 may bebuilt into a column sense line amplifier 4304 that obtains outputsignals from the pixel 4302 via a column sense line 4306.

Further Illustrated Aspects of the Sensing Systems

FIG. 44 shows an exemplary transmit and receive (T_(x)-R_(x)) system4400. For example, the transmit and receive (T_(x)-R_(x)) system 4400may be the same as or similar to one or more of those explained above.In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4400 may include a transmitter 4402 and a receiver4404. In some of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4400 may be configured and arranged to offset thetransmitter 4402 and the receiver 4404 from each other by a distance D.In some of the various embodiments, the receiver 4404 may include a SPAD(single photon avalanche detector) array sensor 4406. In someembodiments, the sensor 4406 may include an active column-gated SPADarray that includes a plurality of columns and a plurality of rows ofSPAD pixels. In some embodiments, the transmit and receive (T_(x)-R_(x))system 4400 may speculatively activate columns of the SPAD array.

In one or more of the various embodiments, the transmitter 4402 may emitnarrowly collimated, one-dimensional, blade-like, photonic bursts(“blades”) 4408. For example, the bursts 4408 may have short durations(e.g., 100 picoseconds). In some of the various embodiments, thetransmitter 4402 may emit the blades 4408 outward at a lateral angle α.In some embodiments, light of these blades 4408 may vertically spreadacross a field of view (e.g., along a Y-direction). In some embodiments,light of these blades 4408 may maintain a sharply focused light bladeedge in a scan direction (e.g., in an X-direction, orthogonal to an edgeof a given one of the blades 4408). In one or more of the variousembodiments, the receiver 4404 may capture reflected blades. In someembodiments, a width of an image of a reflected blade at the sensor 4406may be narrower than a width of a SPAD column in the SPAD array.

In one or more of the various embodiments, the transmitter 4402 mayemploy one-dimensional (e.g., cylindrical) laser collimated opticsapplied to one or more powerful striped edge emitting diode lasersources to produce the blades 4408 via intense bursts while maintaininga sharp edge. For example, the transmitter 4402 may employ a continuousmotion one-dimensional scanning system that includes one or more of aone-dimensional MEMS mirror, a galvanic mirror, a phased array, or aspinning polygonal mirror wheel reflector. In some of the variousembodiments, the transmit and receive (T_(x)-R_(x)) system 4400 mayemploy the offset distance D to ensure that light of blades 4408 emittedat a given scan angle α and reflected from different Z-ranges arecaptured by SPAD pixels in different columns of the SPAD array. In oneor more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4400 may implement a slower transmitter whileemploying blades 4408 as opposed to photon bullets because thetransmitter 4402 may scan the blades 4408 across a horizontal dimensionas opposed to both the horizontal and vertical dimensions.

For example, the blades 4408 may spread vertically across the entirefield of view while traveling toward a first object 4410 and a secondobject 4412. The second object 4412 may be located at a position that isfurther from the transmit and receive (T_(x)-R_(x)) system 4400 than thefirst object 4410 by a distance ΔZ. The second object 4412 may belocated at a position that is more elevated in the field of view thanthe first object 4410 by an incremental elevation angle ΔE. A first raywithin a given blade may reflect off the first object 4410 while asecond ray within the given blade may continue toward and reflect offthe second object 4412. One or more pixels in a first column 4414 thathas a first disparity in the SPAD array may detect photons of the firstray. One or more pixels in a second column 4416 that has a seconddisparity in the SPAD array may detect photons of the second ray. Atleast because of the offset distance D, the transmit and receive(T_(x)-R_(x)) system 4400 may determine that the first object 4410 is ata first distance Z based on the first disparity and may determine thatthe second object 4412 is at a second distance Z based on the seconddisparity (e.g., by employing a lookup table based on one or more ofdisparity or scan angle α such as, for example, explained in furtherdetail below).

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4400 may provide an actively column-gated scanningLIDAR system. In some of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 4400 may employ time-of-flight as a rangingmethod. The transmit and receive (T_(x)-R_(x)) system 4400 may suppressambient light by employing triangular light path geometry as aspatial-temporal filter. As explained above, portions of a reflectedblade may arrive at the receiver 4404 sooner or later depending on aZ-range of an object that reflected the blade portion. Variance intime-of-flight required to cover different distances may also beproportional to a lateral displacement (e.g., as explained above). Asexplained above, the transmit and receive (T_(x)-R_(x)) system 4400 maydetect lateral displacement based on which column contains one or morepixels that capture light of the reflected blade.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4400 may employ a choreographed succession ofjust-in-time activated columns. In some of the various embodiments, thetransmit and receive (T_(x)-R_(x)) system 4400 may successively activatecolumns of SPAD pixels. In some of the various embodiments, the transmitand receive (T_(x)-R_(x)) system 4400 may adjust activation periods ofsuccessively activated columns. In some embodiments, the transmit andreceive (T_(x)-R_(x)) system 4400 may increase the activation periods asthe succession of column activation proceeds. For example, the transmitand receive (T_(x)-R_(x)) system 4400 may match the activation periodsto ever greater periods time-of-flight delays for a given blade, withsuccessively diminishing column position disparity shifts for a givenincrease in Z-range). In one or more of the various embodiments, thetransmit and receive (T_(x)-R_(x)) system 4400 may vary the activationperiods to reduce capture of ambient light by pixels of the SPAD array(e.g., as explained in further detail below).

FIG. 45 shows an exemplary SPAD array sensor 4500 of an exemplarytransmit and receive (T_(x)-R_(x)) system. For example, the transmit andreceive (T_(x)-R_(x)) system may be the same as or similar to one ormore of those explained above. In one or more of the variousembodiments, the sensor 4500 may include an active column-gated SPADarray 4502. In some of the various embodiments, the transmit and receive(T_(x)-R_(x)) system may employ the SPAD array 4502 to provide anaccurate, actively column-gated scanning LIDAR system.

In one or more of the various embodiments, the SPAD array 4502 may havea plurality of columns and a plurality of rows of SPAD pixels. Forexample, the SPAD array sensor 4500 may have 500 columns and 200 rows ofSPAD pixels for a total of 10,000 SPAD pixels. In some of the variousembodiments, each SPAD pixel may have dimensions of five microns by fivemicrons (or six microns by six microns or eight microns by eightmicrons, which is 64 times larger in area than one-micron pixels foundin inexpensive modern rolling shutter cameras). In some embodiments, theSPAD array 4502 may have a height of 1 mm and a width of 2.5 mm.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system may successively activate individual columns of theSPAD array 4502. In some of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system may successively activate each column for aparticular time period at a particular time subsequent to a start timeof the sequential activation. In some embodiments, the transmit andreceive (T_(x)-R_(x)) system may determine, for each individual column,the respective particular time based on a determined offset distancebetween a transmitter and a receiver of the transmit and receive(T_(x)-R_(x)) system (e.g., via a lookup table as explained in furtherdetail below). Additionally or alternatively, the transmit and receive(T_(x)-R_(x)) system may determine, for each individual column, therespective particular time based on a determined scan angle α (e.g., viathe lookup table).

For example, the SPAD array 4502 may capture reflections of a particularphoton blade that reflected off two objects at two respective Z-rangesand two respective elevations (e.g., as explained above with regard toFIG. 44). The transmit and receive (T_(x)-R_(x)) system may activate afirst column 4504 of SPAD pixels at a first time for a first activationperiod. The transmit and receive (T_(x)-R_(x)) system may activate asecond column 4506 of SPAD pixels at a second time for a secondactivation period. The second time may be subsequent to the first timeby a particular time interval At. In one or more of the variousembodiments, the second activation period may exceed the firstactivation period. A first SPAD pixel 4508 in a first row 4510 and thefirst activated column 4504 may trigger a first avalanche 4511responsive to the first SPAD pixel 4508 capturing a first one of thereflections during the first activation period. A second SPAD pixel 4512in a second row 4514 and the second activated column 4506 may trigger asecond avalanche 4515 responsive to the second SPAD pixel 4512 capturinga second one of the reflections during the second activation period.Each row of pixels may communicatively couple to a respective signalline. The first SPAD pixel 4508 may output the first avalanche to afirst signal line 4516. The second SPAD pixel 4512 may output the secondavalanche to a second signal line 4518. A time interval between when thefirst SPAD pixel 4508 outputs the first avalanche to the first signalline 4516 and when the second SPAD pixel 4512 outputs the secondavalanche to the second signal line 4518 may equal or substantiallyequal the particular time period At. The transmit and receive(T_(x)-R_(x)) system may determine that the first avalanche on the firstsignal line 4516 was output from the first activated column 4504 andthat the second avalanche on the second signal line 4518 was output fromthe second activated column 4506 based on one or more of the first time,the second time, or the particular time interval At between the firstand second times. The transmit and receive (T_(x)-R_(x)) system maydetermine the two respective Z-ranges of the two objects based on thedetermined first and second activated columns 4504, 4506 (e.g., asexplained above). The transmit and receive (T_(x)-R_(x)) system maydetermine the two respective elevations of the two objects based on thefirst and second rows 4510, 4514 of the first and second SPAD pixels4508, 4512 (e.g., as explained above).

FIG. 46 illustrates an exemplary choreographed successive SPAD pixelcolumn activation by an exemplary transmit and receive (T_(x)-R_(x))system. For example, the transmit and receive (T_(x)-R_(x)) system maybe the same as or similar to one or more of those explained above. Inone or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system may activate a first column in a SPAD array at afirst time 4600. In some of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system may activate a second column in the SPADarray at a second time 4602. In some embodiments, the transmit andreceive (T_(x)-R_(x)) system may activate the first column for a firsttime period 4604 and may activate the second column for a second timeperiod 4606. Responsive to one or more SPAD pixels in the first columncapturing a sufficient quantity of photons to trigger during the firsttime period 4604, the one or more SPAD pixels in the first column mayavalanche at a third time 4608. Responsive to one or more SPAD pixels inthe second column capturing a sufficient quantity of photons to triggerduring the second time period 4606, the one or more SPAD pixels in thesecond column may avalanche at a fourth time 4610. In some embodiments,the fourth time 4610 may be subsequent to the third time 4608 by aduration At (e.g., as explained above).

FIG. 47 shows an exemplary transmit and receive (T_(x)-R_(x)) system4700 that employs an exemplary series of light blades and an exemplarySPAD array 4702 that captures reflections of the light blades. Forexample, the transmit and receive (T_(x)-R_(x)) system 4700 may be thesame as or similar to one or more of those explained above. In one ormore of the various embodiments, the transmit and receive (T_(x)-R_(x))system 4700 may include a transmitter 4704 and a receiver 4706. In someof the various embodiments, the transmit and receive (T_(x)-R_(x))system 4700 may be configured and arranged to offset the transmitter4704 and the receiver 4706 from each other by a distance D. In someembodiments, the transmit and receive (T_(x)-R_(x)) system 4700 mayemploy the offset distance D to measure one or more Z-ranges of one ormore objects that reflect light to the receiver 4706.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4700 may select a sequence of columns in the SPADarray 4702. In some of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4700 may determine that differences betweendisparities of successive columns in the sequence of columns varies as afunction of one or more of the offset distance D, respective Z-rangesthat corresponds to each of the successive columns, and a scan angle α.For example, the transmit and receive (T_(x)-R_(x)) system 4700 mayassociate, for each given SPAD pixel in a given row of pixels, uniquenarrow (e.g., telescopic) sections of a field of view of the transmitand receive (T_(x)-R_(x)) system 4700 that the given SPAD pixel observes(e.g., a narrow section, such as a horizontal slice, of a trajectory ofa photon blade). In some embodiments, the transmit and receive(T_(x)-R_(x)) system 4700 may determine that photons captured by thegiven SPAD pixel indicate a reflection off an object in the narrowsection associated with the given pixel (e.g., based on self-reportingby active smart pixels such as the avalanches that the SPAD pixelsoutput to reveal a voxel position) as explained above.

For example, the transmit and receive (T_(x)-R_(x)) system 4700 maydetermine, for each given Z-range in a sequence of potential Z-ranges,that light of a given blade emitted at a particular time may reflect offa given object at the given Z-range and arrive at the SPAD array 4702 ata respective expected arrival time (e.g., based on time-of-flight asexplained above). In some embodiments, the transmit and receive(T_(x)-R_(x)) system 4700 may also determine, for each given Z-range inthe sequence of potential Z-ranges, which one or more columns of theSPAD array 4702 contains one or more pixels that may capture the lightof the given blade emitted at the particular time and reflected off thegiven object. In some embodiments, the transmit and receive(T_(x)-R_(x)) system 4700 may make one or more of these determinationsfor each of a plurality of scan angles α. The transmit and receive(T_(x)-R_(x)) system 4700 may selectively include the determined one ormore columns in a sequence of columns.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4700 may associate a first column 4708 in thesequence of columns with a first Z-range 4710 in the sequence ofZ-ranges, associate a second column 4712 in the sequence of columns witha second Z-range 4714 in the sequence of Z-ranges, associate a thirdcolumn 4716 in the sequence of columns with a third Z-range 4718 in thesequence of Z-ranges, associate a fourth column 4720 in the sequence ofcolumns with a fourth Z-range 4722 in the sequence of Z-ranges,associate a fifth column 4724 in the sequence of columns with a fifthZ-range 4726 in the sequence of Z-ranges, and associate a sixth column4728 in the sequence of columns with a sixth Z-range 4730 in thesequence of Z-ranges. An expected time of arrival at one or more pixelsin the first column 4708 for a reflection of light of a given blade offa given object at the first Z-range 4710 may precede respective expectedtimes of arrival for each subsequent column in the sequence of columns.A Z-range increment from each given Z-range in the sequence of potentialZ-ranges to an immediately subsequent Z-range in the sequence ofpotential Z-ranges may be equal to a Z-range increment from animmediately prior Z-range in the sequence of potential Z-ranges to thegiven Z-range. In contrast, a disparity increment between a firstsuccessive pair of columns in the sequence of columns may be differentfrom a disparity increment between a second successive pair of columnsin the sequence of columns.

FIG. 48 illustrates an exemplary SPAD array 4800 of an exemplarytransmit and receive (T_(x)-R_(x)) system. For example, the transmit andreceive (T_(x)-R_(x)) system may be the same as or similar to one ormore of those explained above. In one or more of the variousembodiments, the transmit and receive (T_(x)-R_(x)) system may determinethat, for columns that receive reflections off objects at equallyincremented Z-ranges, differences between disparities of the columns areunequal (e.g., as shown in FIG. 48). In some of the various embodiments,the transmit and receive (T_(x)-R_(x)) system may determine that thedifferences between these disparities vary as a function of one or moreof the offset distance D, a Z-range that corresponds to each of thesuccessive columns, and a scan angle α (e.g., as explained above).

Returning to FIG. 44, the transmit and receive (T_(x)-R_(x)) system 4400may determine that, for a given scan angle α, a receive angle βincreases as the Z-range increases (e.g., from β₁ at the Z-range of thefirst object 4410 to β₂ at the Z-range of the second object 4412). Inone or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4400 may determine that, for a given Z-range, thereceive angle β decreases as the scan angle α increases. In some of thevarious embodiments, the sensor 4406 may perceive the disparity in theimages caused by the reflections of the blades as diminishing as lightfrom the blades reflects off objects at greater Z-ranges.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4400 may speculatively activate each column of theSPAD array sensor 4406. In some embodiments, the transmit and receive(T_(x)-R_(x)) system 4400 may determine, for each of a plurality ofpotential Z-ranges of a given object, that light of a given bladeemitted at a particular time may reflect off the given object and arriveat the SPAD array sensor 4406 at a respective expected arrival time(e.g., based on time-of-flight as explained above). In some embodiments,the transmit and receive (T_(x)-R_(x)) system may also determine, foreach of the plurality of potential Z-ranges, which one or more columnsof the SPAD array sensor 4406 contains one or more pixels that shouldcapture the light of the given blade emitted at the particular time andreflected off the given object. In some embodiments, the transmit andreceive (T_(x)-R_(x)) system 4400 may make one or more of thesedeterminations for each of a plurality of scan angles α. In one or moreof the various embodiments, the transmit and receive (T_(x)-R_(x))system 4400 may speculatively activate a given column of the SPAD arraysensor 4406 immediately prior to the expected arrival time for the lightof the given blade that one or more pixels of the given column shouldcapture from the reflection off the given object at the potentialZ-range that corresponds to the given column. In some of the variousembodiments, the transmit and receive (T_(x)-R_(x)) system 4400 mayspeculatively activate the given column by setting each SPAD pixel inthe given column to avalanche as soon as the SPAD pixel captures asufficient quantity of photons (e.g., as few as one). In someembodiments, the transmit and receive (T_(x)-R_(x)) system 4400 may setthe SPAD pixels in the speculatively activated column by brieflyconnecting a high-voltage reverse biased signal to each SPAD pixel inthe speculatively activated column at the same time. For example, thetransmit and receive (T_(x)-R_(x)) system 4400 may apply a reversebiased signal that has a voltage that causes the SPAD pixels to reachGeiger mode. In Geiger mode, an avalanche diode in each SPAD pixel may,responsive to capturing the sufficient quantity of photons, output astrong instantaneous signal (e.g., up to 10,000 electrons). The stronginstantaneous signal may provide an easy to detect, low latency, lowjitter time signal. The signal may have a temporal accuracy in the rangeof approximately 100 ps.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4400 may speculatively activate columns in a rangeof columns (e.g., 100 to 200 columns) that corresponds to a Z-range ofinterest. In some of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4400 may emit a plurality of blades in quicksuccession (e.g., emitting one or more blades prior to capture by theSPAD array sensor of a reflection of a previously emitted blade). Insome embodiments, the transmit and receive (T_(x)-R_(x)) system 4400 mayemit multiple blades per predicted range of time of flight for a Z-rangeof interest. For example, the transmit and receive (T_(x)-R_(x)) system4400 may emit ten blades at the same scan angle α, each blade having aduration of 10 ps and spaced 100 ns from an immediately prior emittedblade. The transmit and receive (T_(x)-R_(x)) system 4400 mayspeculatively sequentially activate each column that corresponds to aZ-range in the Z-range of interest every 100 ns. For this example, aSPAD array sensor that has 200 row sense lines may support up to a 10Mhz burst output data rate, resulting in a total system peak data rateof 2 billion voxels per second. The transmit and receive (T_(x)-R_(x))system 4400 may permit employing large SPAD pixels while providing botha small sensor footprint and high accuracy (e.g., via subnanosecondtiming in time-of-flight measurements).

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4400 may dynamically adjust one or more ofthresholds, emission patterns, column activation controls, or gain. Insome of the various embodiments, the transmit and receive (T_(x)-R_(x))system 4400 may employ one or more of those dynamic adjustments tofilter or prioritize one or more data streams (e.g., provideprogrammable data priority and/or data confidence levels). In someembodiments, the transmit and receive (T_(x)-R_(x)) system 4400 may makeone or more of those dynamic adjustments based on one or more settingssuch as range selection, foveation, object locking, or Z-locking (e.g.,as explained above). For example, the transmit and receive (T_(x)-R_(x))system 4400 may vary gain applied to each individual column based on aduration of activation for the individual column. In some of the variousembodiments, the transmit and receive (T_(x)-R_(x)) system 4400 maybalance bias of SPAD pixels in a given column with duration ofactivation for the given column (e.g., gain may be inverselyproportional to activation duration: decrease gain for longer activationduration or increase gain for shorter activation duration) to manage alikelihood of one or more SPAD pixels in the given column spontaneouslyavalanching (e.g., responsive to ambient light) and thereby providing afalse detection. For example, the transmit and receive (T_(x)-R_(x))system 4400 may bias SPAD pixels in the given column at 20 volts whenactivating the given column for 10 ns. The transmit and receive(T_(x)-R_(x)) system 4400 may decrease the bias from 20 volts whenactivating the given column for a longer activation duration or mayincrease the bias from 20 volts when activating the given column for ashorter activation duration. Additionally or alternative, the transmitand receive (T_(x)-R_(x)) system 4400 may adjust gain applied to columnsbased on a quantity of SPAD pixels in each column. In some embodiments,the transmit and receive (T_(x)-R_(x)) system 4400 may apply a largergain to columns of a SPAD array sensor that has a high quantity of rowsas compared to a smaller gain that the transmit and receive(T_(x)-R_(x)) system 4400 may apply to columns of a SPAD array sensorthat has a lower quantity of rows. In one or more of the variousembodiments, the transmit and receive (T_(x)-R_(x)) system 4400 mayactivate each column for 10-50 ns.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4400 may initiate a subsequent speculativesequential activation prior to conclusion of a prior speculativesequential activation. In some embodiments, the transmit and receive(T_(x)-R_(x)) system 4400 may speculatively activate columns in aplurality of ranges of columns in parallel (e.g., a plurality of rangesthat correspond to a plurality of Z-ranges of interest). For example,the transmit and receive (T_(x)-R_(x)) system 4400 may apply distinctranges of activation times for each of the plurality of ranges ofcolumns.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4400 may dynamically choreograph the speculativesequential activation. In some of the various embodiments, the transmitand receive (T_(x)-R_(x)) system 4400 may dynamically choreograph thespeculative sequential activation based on the scan angle α. In someembodiments, the transmit and receive (T_(x)-R_(x)) system 4400 mayadjust activation duration for one or more columns in a range of columnsthat the transmit and receive (T_(x)-R_(x)) system 4400 may sequentiallyactivate. In some embodiments, the transmit and receive (T_(x)-R_(x))system 4400 may adjust the range of columns. For example, the transmitand receive (T_(x)-R_(x)) system 4400 may determine, for each of aplurality of Z-ranges, one or more of new expected times of flight ornew expected disparities for one or more columns. In one or more of thevarious embodiments, such values could be precalculated. For example,the transmit and receive (T_(x)-R_(x)) system 4400 may make one or moreof these determinations during a calibration routine and store one ormore results in a fast lookup table (e.g., one or more of the lookuptables explained above).

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4400 may determine that an object that reflectslight from a blade may have a flat surface based on a quantity of rowsof pixels in an activated column that capture the reflected light (e.g.,all rows in the activated column may have a SPAD pixel that captured thereflected light). In some of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 4400 may determine that the object hasmultiple sloped surfaces based on SPAD pixels in different activatedcolumns capturing the reflected light.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4400 may, in the case of a direct time-of-flightmethod, have a Z-resolution that is a function of arrival time of bladepulses. In contrast, the transmit and receive (T_(x)-R_(x)) system 4400may, in the case of deriving a Z-range via triangulation, have aZ-resolution that is a function of column position (e.g., accuracy ofrelative displacement, pixel disparity and the X-direction, or thelike). In some of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4400 may determine the Z-range via time of flight.In some embodiments, the transmit and receive (T_(x)-R_(x)) system 4400may employ additional triangular light path geometry as aspatial-temporal filter to suppress ambient light.

In one or more of the various embodiments, the speculative sequentialactivation may provide 10-100 times greater performance thancapabilities of existing systems by Velodyne™ or Quanergy™ (e.g.,systems by Velodyne™ or Quanergy™ that focus on one or more ofautonomous vehicular LIDAR or three-dimensional perception domain). Insome of the various embodiments, the transmit and receive (T_(x)-R_(x))system 4400 may operate one or more various asynchronous sensor arrays(e.g., asynchronous camera or twitchy pixels) in place of the SPAD arraysensor 4406 while employing the above-explained spatially filtered linescan and time-of-flight detection.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4400 may emit a given light blade that containsmultiple colors. In some of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 4400 may employ parallel or sequentialfilterless color LIDAR. In some embodiments, the SPAD array sensor 4406may be sensitive for a plurality of light frequencies (e.g., from NIRthrough UV). In some embodiments, the transmit and receive (T_(x)-R_(x))system 4400 may operate in total ambient darkness. For example, byrapidly changing illumination wavelengths of blade pulses (e.g.,switching between primary colors), the transmit and receive(T_(x)-R_(x)) system 4400 may determine color and hue of athree-dimensional surface that the transmit and receive (T_(x)-R_(x))system 4400 scans.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4400 may, by employing the above-explainedsequential active column-gating, reduce ambient light that a given SPADpixel in an activated column experiences to one millionth of ambientlight the SPAD pixel would otherwise experience (e.g., a reduction of100,000 lux from to 0.1 lux or sunlight interference to moonlightinterference). In some of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 4400 may improve eye safety by emittinglower instantaneous power density of the blades as compared to emissionsof conventional systems.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4400 may, as explained above, adjust activationperiods of successive columns to match incrementing time-of-flightdelays as the Z-range increases and incremental column disparitydecreases. In some of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4400 may, by adjusting the activation periods,ensure detection of reflected light by one or more pixels andinstantaneous range measurement along line of sight of the one or morepixels. As explained above, the transmit and receive (T_(x)-R_(x))system 4400 may, from a single blade pulse, provide a high quantity ofinstant separate voxel measurements (e.g., 200) within a short timeperiod (e.g., 1 μs) that can be as high as a vertical pixel count in agiven column of a sensor. In some embodiments, the transmit and receiveT_(x)-R_(x)) system 4400 may provide a data rate of 200 million voxelsper second with a SPAD array (e.g., far superior to expensive LIDARsystems on the market such as those from Velodyne™ or “solid state”LIDAR systems such as those from Quanergy ™). In some embodiments, thetransmit and receive (T_(x)-R_(x)) system 4400 may set a maximum photonroundtrip time of flight to 1,000 ns or 1 μs, thereby capping observedZ-range to 500 ft. In contrast, in a conventional SPAD detector, onlyone observation per microsecond may be supported.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4400 may, while employing time-of-flightmeasurements of Z-range, have a Z-range resolution that is a function ofarrival time of one or more pulses. In contrast, the transmit andreceive (T_(x)-R_(x)) system 4400 may, while employing triangulation,have a Z-range resolution that is a function of resolution of columnposition (e.g., accuracy of relative displacement or pixel disparity inthe X-direction). In some of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 4400 may, while employing theabove-explained sequential active column-gating with blade pulses, havea Z-range resolution that is a function of one or more of columnposition. In some embodiments, the transmit and receive (T_(x)-R_(x))system 4400 may, while employing time-of-flight measurements via theabove-explained sequential active column-gating with blade pulses, havea resolution in the horizontal X-direction that is a function of bothsharpness of the light blade and accuracy of determining the scanningangle α. For example, the transmit and receive (T_(x)-R_(x)) system 4400may determine the scan angle α via anticipatory prediction orinterpolation ex-post (e.g., motion interpolation) as explained withregard to FIG. 4B of U.S. Pat. No. 8,282,222. In some of the variousembodiments, the transmit and receive (T_(x)-R_(x)) system 4400 may,while employing the above-explained sequential active column-gating withblade pulses have a vertical Y-direction resolution that is a functionof a quality of SPAD row optics and row resolution of the SPAD sensor4406. In one or more of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 4400 may employ an activation period of onetenth of a nanosecond (100 ps) for a given column and thereby resolve along-distance object down to one centimeter.

In one or more of the various embodiments, the transmit and receive(T_(x)-R_(x)) system 4400 may employ each blade as a scan line asexplained above. In some of the various embodiments, the transmit andreceive (T_(x)-R_(x)) system 4400 may employ the blades to providesharpness of a sequential pixel scanning system and breadth of a flashLIDAR system as explained above. For example, sequential pixel scanningsystems may employ photon pin pricks with one individual pinprick foreach individual voxel. Sequential pixel scanning systems requireextremely fast scan rates. As another example, flash LIDAR systems mayemploy no scanner and, instead, may require extremely high instantaneouspower sources to ensure a sufficient instantaneous photon count withstatistical certainty for each pixel in a sensor array. In contrast toboth of these examples, the transmit and receive (T_(x)-R_(x)) system4400 may emit blade pulses that vertically cover all rows for a givencolumn in the sensor 4406, thereby enabling the transmit and receive(T_(x)-R_(x)) system 4400 to employ a slower scan rate (e.g., 100 Hz for100 frames per second), scan in a single direction, and employ largertime gaps between pulses while providing a high voxel throughput. Insome embodiments, the transmit and receive (T_(x)-R_(x)) system 4400 mayemit 2,000 blade pulses laterally spread across a 30 degree field ofview in 4 ms while providing an HD lateral voxel resolution at up to 250frames per second. For example, the transmit and receive (T_(x)-R_(x))system 4400 may employ 250 lines of vertical resolution to provide a rawvoxel count of 125 million voxels per second (e.g., 250 lines×250 framesper second×2,000 blade pulses).

It will be understood that each block of the flowchart illustrations,and combinations of blocks in the flowchart illustrations, (or actionsexplained above with regard to one or more systems or combinations ofsystems) can be implemented by computer program instructions. Theseprogram instructions may be provided to a processor to produce amachine, such that the instructions, which execute on the processor,create means for implementing the actions specified in the flowchartblock or blocks. The computer program instructions may be executed by aprocessor to cause a series of operational steps to be performed by theprocessor to produce a computer-implemented process such that theinstructions, which execute on the processor to provide steps forimplementing the actions specified in the flowchart block or blocks. Thecomputer program instructions may also cause at least some of theoperational steps shown in the blocks of the flowcharts to be performedin parallel. Moreover, some of the steps may also be performed acrossmore than one processor, such as might arise in a multi-processorcomputer system. In addition, one or more blocks or combinations ofblocks in the flowchart illustration may also be performed concurrentlywith other blocks or combinations of blocks, or even in a differentsequence than illustrated without departing from the scope or spirit ofthe invention.

Additionally, in one or more steps or blocks, may be implemented usingembedded logic hardware, such as, an Application Specific IntegratedCircuit (ASIC), Field Programmable Gate Array (FPGA), Programmable ArrayLogic (PAL), or the like, or combination thereof, instead of a computerprogram. The embedded logic hardware may directly execute embedded logicto perform actions some or all of the actions in the one or more stepsor blocks. Also, in one or more embodiments (not shown in the figures),some or all of the actions of one or more of the steps or blocks may beperformed by a hardware microcontroller instead of a CPU. In at leastone embodiment, the microcontroller may directly execute its ownembedded logic to perform actions and access its own internal memory andits own external Input and Output Interfaces (e.g., hardware pins and/orwireless transceivers) to perform actions, such as System On a Chip(SOC), or the like.

The above specification, examples, and data provide a completedescription of the manufacture and use of the composition of theinvention. Since many embodiments of the invention can be made withoutdeparting from the spirit and scope of the invention, the inventionresides in the claims hereinafter appended.

What is claimed as new and desired to be protected by Letters Patent ofthe United States is:
 1. A method for measuring a three-dimensionalrange to a target, wherein one or more processors execute instructionsthat perform actions of the method, comprising: employing a transmitterto transmit light toward the target and a receiver to detect one or morereflections of the transmitted light, wherein the receiver is physicallyoffset separate from the transmitter; determining one or moreanticipated arrival times of the one or more reflections based on one ormore departure times of the transmitted light and a length of thephysical offset; energizing separately different portions of a pluralityof pixels for the receiver based on the one or more anticipated arrivaltimes; and employing the different portions of energized pixels todetect an amount of photons from the one or more reflections of thetransmitted light, wherein a disparity of a positional offset of one ormore pixels in the receiver relative to a predetermined position in thereceiver is used to provide a three-dimensional measurement of the rangeof the target.